<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>R-bloggers</title>
	<atom:link href="https://www.r-bloggers.com/feed/" rel="self" type="application/rss+xml" />
	<link>https://www.r-bloggers.com</link>
	<description>R news and tutorials contributed by hundreds of R bloggers</description>
	<lastBuildDate>Mon, 16 Nov 2026 09:05:50 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=5.5.24</generator>

<image>
	<url>https://i0.wp.com/www.r-bloggers.com/wp-content/uploads/2016/08/cropped-R_single_01-200.png?fit=32%2C32&#038;ssl=1</url>
	<title>R-bloggers</title>
	<link>https://www.r-bloggers.com</link>
	<width>32</width>
	<height>32</height>
</image> 
<site xmlns="com-wordpress:feed-additions:1">11524731</site>	<item>
		<title>smooth in Python: TBATS implementation</title>
		<link>https://www.r-bloggers.com/2026/11/smooth-in-python-tbats-implementation/</link>
		
		<dc:creator><![CDATA[Ivan Svetunkov]]></dc:creator>
		<pubDate>Mon, 16 Nov 2026 09:05:50 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://openforecast.org/?p=4717</guid>

					<description><![CDATA[<p>TBATS is a popular exponential smoothing model for data with complex seasonality: long seasonal periods, fractional ones (e.g. 52.18 weeks in a year, as I discussed in my previous post), and several of them at once. It models seasonality with harmonics and includes a Box-Cox transformation, ETS level and trend, ...</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/smooth-in-python-tbats-implementation/">smooth in Python: TBATS implementation</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://openforecast.org/2026/11/16/smooth-in-python-tbats-implementation/"> Archives R - OpenForecast</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<p>TBATS is a popular exponential smoothing model for data with complex seasonality: long seasonal periods, fractional ones (e.g. 52.18 weeks in a year, as I <a href="https://openforecast.org/2026/11/09/the-problem-with-weekly-seasonality/" rel="nofollow" target="_blank">discussed in my previous post</a>), and several of them at once. It models seasonality with harmonics and includes a Box-Cox transformation, ETS level and trend, and ARMA errors (<a href="https://doi.org/10.1198/jasa.2011.tm09771" rel="nofollow" target="_blank">De Livera, Hyndman &#038; Snyder, 2011</a>).</p>
<p>But the original formulation has an issue. Its ARMA part sits in the error term, which then updates all the other states, including the level. So, the level and the ARMA compete for the same dynamics, and as the AR parameter approaches one, the permanent effect of a shock on the level grows without limit.</p>
<p>Claude and I have now implemented TBATS in the smooth package for Python (and R), reusing the core of ADAM. Without ARMA, it is exactly the original model (to rounding error). But ARMA is now a separate component: the level keeps its share of every shock, and the effect of ARMA decays over time, as it should.</p>
<p>The ADAM framework also brings several features that other implementations don’t have:</p>
<ol>
<li>Explanatory variables, e.g. promotions or prices;</li>
<li>Non-normal distributions, selected automatically;</li>
<li>Treatment of missing values (e.g. weeks with stockouts);</li>
<li>An occurrence model for intermittent demand.</li>
</ol>
<p>These are not bells and whistles: real demand data has promotions, gaps and zeros, and a model that cannot handle them pushes you into pre-processing hacks.</p>
<p>Here is an example on weekly sales with two promotional campaigns:</p>
<pre>from fcompdata import PromoData
from smooth import TBATS

model = TBATS(lags=[1, 52], h=13, holdout=True)
model.fit(PromoData.y, X=PromoData.xreg)
print(model)
fc = model.predict(h=13, interval=&quot;prediction&quot;, level=0.95)</pre>
<p>The function selects the number of harmonics, the trend, the ARMA orders and the distribution automatically, and the effects of the promotions are estimated jointly with everything else.</p>
<p>Does it work? I compared it with <code>forecast::tbats()</code> in R and <code>AutoTBATS</code> from <code>statsforecast</code> for Python (<a href="https://entrylib.org/p/2026.000007" rel="nofollow" target="_blank">paper</a>), here are the main findings:</p>
<ul>
<li>On 414 hourly series from M4, smooth was more accurate (mean RMSSE of 0.829 vs 0.909 and 1.063), its 95% prediction intervals had better coverage, and it was about 1.7 times faster than <code>forecast::tbats()</code>.</li>
<li>On 370 hourly series of electricity demand, the accuracy was a tie (0.795 vs 0.799 and 0.795), but smooth had better coverage and was again faster.</li>
<li>Also, when I switched to the Normal distribution, the model became even faster, but its 95% intervals covered fewer values than they should, so the automatic selection of distribution seems to help.</li>
</ul>
<p>The table in this post summarises the results. The details are <a href="https://entrylib.org/p/2026.000007" rel="nofollow" target="_blank">here</a>.</p>
<p>Try it: <code>pip install smooth</code><br />
<a href="https://github.com/openforecast-org/smooth/wiki/TBATS" rel="nofollow" target="_blank">Documentation</a></p>
<p>Message <a href="https://openforecast.org/2026/11/16/smooth-in-python-tbats-implementation/" rel="nofollow" target="_blank">smooth in Python: TBATS implementation</a> first appeared on <a href="https://openforecast.org/" rel="nofollow" target="_blank">OpenForecast</a>.</p>

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://openforecast.org/2026/11/16/smooth-in-python-tbats-implementation/"> Archives R - OpenForecast</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/smooth-in-python-tbats-implementation/">smooth in Python: TBATS implementation</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404913</post-id>	</item>
		<item>
		<title>Interim Analysis: My Messy Notes</title>
		<link>https://www.r-bloggers.com/2026/11/interim-analysis-my-messy-notes/</link>
		
		<dc:creator><![CDATA[r on Everyday Is A School Day]]></dc:creator>
		<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://www.kenkoonwong.com/blog/interim-analysis/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
Why can’t we just peek at our data with α = 0.05 every time? 🤔 Gotta see it to believe it! Simulating the inflated error, seeing why it’s not Bonferroni, and rebuilding O’Brien-Fleming &#038; Lan-DeMets boundaries to match gsDesign. Hope it’s not an il...</div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/interim-analysis-my-messy-notes/">Interim Analysis: My Messy Notes</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://www.kenkoonwong.com/blog/interim-analysis/"> r on Everyday Is A School Day</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<blockquote>
<p>Why can’t we just peek at our data with α = 0.05 every time? <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f914.png" alt="🤔" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Gotta see it to believe it! Simulating the inflated error, seeing why it’s not Bonferroni, and rebuilding O’Brien-Fleming &#038; Lan-DeMets boundaries to match gsDesign. Hope it’s not an illusion of learning <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f605.png" alt="😅" class="wp-smiley" style="height: 1em; max-height: 1em;" /></p>
</blockquote>
<p><img src="https://i0.wp.com/www.kenkoonwong.com/blog/interim-analysis/interim-analysis.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"></p>




<h2 id="motivations">Motivations
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#motivations" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<p>I’ve always heard “interim analysis” thrown around in trial papers but never really understood what’s happening under the hood. Why can’t we just peek at 1.96 every time? And isn’t this just Bonferroni? Turns out it’s not! Let’s dig in and simulate our way to understanding.</p>




<h2 id="objectives">Objectives:
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#objectives" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<ul>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#what" rel="nofollow" target="_blank">What Is Interim Analysis?</a></li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#obrien" rel="nofollow" target="_blank">What’s O’Brien-Fleming Method?</a></li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#code" rel="nofollow" target="_blank">How To Perform It?</a></li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#scratch" rel="nofollow" target="_blank">How To Do It From Scratch?</a>
<ul>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#obrien-method" rel="nofollow" target="_blank">O’Brien-Fleming Method</a></li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#lan" rel="nofollow" target="_blank">Lan De Mat &#038; Obrien Fleming Method</a></li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#compare" rel="nofollow" target="_blank">Comparison</a></li>
</ul>
</li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#opportunities" rel="nofollow" target="_blank">Opportunities For Improvement</a></li>
<li>
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#lessons" rel="nofollow" target="_blank">Lessons Learnt</a></li>
</ul>




<h2 id="what">What Is Interim Analysis?
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#what" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<p>An interim analysis is a planned look at the data half way through a study, before all participants have been enrolled or followed up. They do it for practical and ethical reasons: if a treatment is clearly working, or clearly harmful, it isn’t right to keep randomizing people to the worse option, and if it’s clearly going nowhere, they can save time and resources by stopping early. The catch is that every extra look is another chance to find a “significant” result purely by luck. Peeking twice at the usual 5% threshold pushes the false-positive rate up. Interim analysis methods, such as O’Brien-Fleming boundaries, fix this by setting a stricter bar at the early looks so that the overall error rate stays at 5%.</p>
<p>Now you know me, I’m the see it to believe kind of person. Let’s try to simulate this with multiple trials and see if that’s true</p>
<pre>library(tidyverse)
n &lt;- 500
prob &lt;- 0.5
peek &lt;- c(n/2,n)
zscore_vec &lt;- c()
trial &lt;- 100000
count_false &lt;- vector(mode = &quot;numeric&quot;, length = trial)

for (trial_i in 1:trial) {
  # for reproducibility
  set.seed(trial_i)
  
  # get sim data
  control &lt;- rbinom(n,1,prob)
  rx &lt;- rbinom(n,1,prob)
  
  # look at when n is 250, and when n is 500
  for (i in 1:length(peek)) {
    m  &lt;- peek[i]
    pA &lt;- mean(rx[1:m])
    pB &lt;- mean(control[1:m])
    p_pool &lt;- (pA + pB) / 2
    se &lt;- sqrt(p_pool * (1 - p_pool) * (2 / m))
    
    # calculate z score
    zscore_vec[i] &lt;- abs(pA - pB) / se
  }
  # record zscore if &gt; 1.96 (~pval 0.05)
  if (any(zscore_vec &gt; 1.96)) { count_false[trial_i] &lt;- 1 }
}

# what percent of family-wise error?
(val &lt;- count_false |&gt; mean())

## [1] 0.08952
</pre><p>Lots of code, I hope some of the comments on the codes are self-explanatory. So we basically simulated a no difference result of 2 groups. We used zscore of 1.96 which is our 5% alpha to see if any of these z scores were above the threshold (it’s not supposed to, remember, there is no difference in this truth!). We expect it to be ~5% but we got 8.952% instead! Wow! That’s 3.952% higher than our threshold, not good! <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f914.png" alt="🤔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></p>
<blockquote>
<p>Note to self: The confusion of false postive/discovery/family-wise examples. If you test 90 placebo and 10 real rx and get 15 “significants”, 5 of them wrong: the <code>false positive rate</code> is 5/90 (how often placebos get credit) aka (1 &#8211; specificity), the <code>false discovery rate</code> is 5/15 (how many of your hits are wrong) aka (1 &#8211; ppv [aka precision]), and the <code>family-wise error rate</code> is the chance of getting even one false hit across the whole set, which is what interim analysis controls across looks.</p>
</blockquote>
<p>wow, doesn’t that remind us of something here? Multiplicity, yea? Maybe, kind of. Now, what if we’re being conservative like bonferonni and split that 5% twice? and see how much false positive did we get?</p>
<pre>count_false &lt;- vector(mode = &quot;numeric&quot;, length = trial)

for (trial_i in 1:trial) {
  # for reproducibility
  set.seed(trial_i)
  
  # get sim data
  control &lt;- rbinom(n,1,prob)
  rx &lt;- rbinom(n,1,prob)
  
  # look at when n is 250, and when n is 500
  for (i in 1:length(peek)) {
    m  &lt;- peek[i]
    pA &lt;- mean(rx[1:m])
    pB &lt;- mean(control[1:m])
    p_pool &lt;- (pA + pB) / 2
    se &lt;- sqrt(p_pool * (1 - p_pool) * (2 / m))
    
    # calculate z score
    zscore_vec[i] &lt;- abs(pA - pB) / se
  }
  # record zscore if &gt; 2.24, two tails with two looks (divide 5% to 2)
  if (any(zscore_vec &gt; 2.24)) { count_false[trial_i] &lt;- 1 }
}

# what percent of family wise error rate?
(val &lt;- count_false |&gt; mean())

## [1] 0.04093
</pre><p>What !?!?! 4.093% !?! It’s a bit lower than expected and we may be wasting 0.907% of our threshold! This is also not good! If there truly is a signal, we may be missing it because we are being too conservative. So, what if we use the O’Brien-Fleming method to set our threshold? Let’s see how that works out.</p>




<h2 id="obrien">What’s O’Brien-Fleming Method?
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#obrien" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<p>O’Brien-Fleming is a way of setting stopping boundaries for interim analyses, and it’s the most widely used method in clinical trials. The idea is to be very strict early and lenient late. At the first look, when you have little data, you need overwhelming evidence to stop, a z-score of about 2.80 (pval = 0.005110261) with two looks. By the final analysis, the bar drops back to about 1.98 (pval = 0.04770353), barely above the usual 1.96. Because so little alpha is spent early, the trial almost never stops on a fluke, and the final analysis keeps nearly all of its power. <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f64c.png" alt="🙌" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Let’s take a look of its family wise error rate like before</p>
<pre>count_false &lt;- vector(mode = &quot;numeric&quot;, length = trial)

for (trial_i in 1:trial) {
  # for reproducibility
  set.seed(trial_i)
  
  # get sim data
  control &lt;- rbinom(n,1,prob)
  rx &lt;- rbinom(n,1,prob)
  
  # look at when n is 250, and when n is 500
  for (i in 1:length(peek)) {
    m  &lt;- peek[i]
    pA &lt;- mean(rx[1:m])
    pB &lt;- mean(control[1:m])
    p_pool &lt;- (pA + pB) / 2
    se &lt;- sqrt(p_pool * (1 - p_pool) * (2 / m))
    
    # calculate z score
    zscore_vec[i] &lt;- abs(pA - pB) / se
  }
  # record zscore if &gt; O&#39;Brien-Fleming, two looks, notice the zscores are no longer 1.96
  bounds &lt;- c(2.80, 1.98) 
  if (any(zscore_vec &gt; bounds)) {count_false[trial_i] &lt;- 1}
}

# what percent of family wise error?
(val &lt;- count_false |&gt; mean())

## [1] 0.0489
</pre><p>Alright! That’s much better! <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f44d.png" alt="👍" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Closer to the 5%. As I was learning this, I just realized that this is very different from multiplicity because the second look technically is correlated to the first look because it looks at the same data again! The two looks share half their data, so their z-scores have a correlation of √0.5 ≈ 0.71. Bonferroni treats them as if they were barely related, so it over-adjusts. <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f914.png" alt="🤔" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Wow, that totally makes sense now! Ok, now how do we get sample calculation + obrien fleming method with <code>gsDesign</code>? 
<a href="https://www.kenkoonwong.com/blog/interim-analysis/#scratch" rel="nofollow" target="_blank">Take note below, the history of obrien-fleming and Lan De Mat</a></p>




<h2 id="code">How To Perform It?
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#code" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<p>Let’s say we want to design a RCT of cefdinir vs cephalexin. Looking at 
<a href="https://academic.oup.com/ofid/article/12/10/ofaf501/8277239?login=false" rel="nofollow" target="_blank">this study</a>, without adjustment, cefdinir had a 23.4% treatment failure compared to cephalexin 12.5%. With 80% power, what sample size, what’s the upper boundary of z score we need if we want 3 looks on our interim analyses?</p>
<pre>library(gsDesign)
p1 &lt;- 0.234
p2 &lt;- 0.125

# fixed design: total sample size, both groups together
(n_fix &lt;- nBinomial(p1 = p1, p2 = p2, alpha = 0.025, beta = 0.2))

## [1] 386.8202
</pre><p>Total sample size needed is 386.820172, will have to divide this by 2. Let’s have gsDesign tell us the upper boundary zscore and when to look.</p>
<pre># group sequential: 3 equally spaced looks, O&#39;Brien-Fleming spending
d &lt;- gsDesign(k = 3, test.type = 2, alpha = 0.025, beta = 0.2,
              n.fix = n_fix, sfu = sfLDOF)

sample &lt;- d$n.I / 2           
bound &lt;- d$upper$bound           
pval &lt;- 2 * (1 - pnorm(d$upper$bound))  
(table &lt;- tibble(number_per_group=sample,zscore_needed=bound,p_value=pval))

## # A tibble: 3 × 3
##   number_per_group zscore_needed  p_value
##              &lt;dbl&gt;         &lt;dbl&gt;    &lt;dbl&gt;
## 1             65.3          3.71 0.000207
## 2            131.           2.51 0.0120  
## 3            196.           1.99 0.0463
</pre><p>Per group, we will need the above numbers to take a peek. Hooray! Now we kind of understand the surface of the goal and why we need to do this so we don’t inflate our family-wise error rate. Now, let’s try to do this from scratch and see if we can get the same numbers as <code>gsDesign</code> did. Take note that this time we will be using the first simulate DGP with no diff, not the cefdinir study.</p>




<h2 id="scratch">How To Do It From Scratch?
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#scratch" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<p>So who came up with what? <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f914.png" alt="🤔" class="wp-smiley" style="height: 1em; max-height: 1em;" /> O’Brien and Fleming came first (1979). Their idea: be super strict early when data is thin, and relax at the end. Every boundary is just one number c divided by √t, and you search for the c that keeps the family-wise error at 5%. Simple, but you had to fix when you’d look ahead of time. Then Lan and DeMets came along (1983) and said, “great idea, but what if we can’t stick to the schedule?” So they made the spending function: decide how much of the 5% you’ve used up at any point, and each look only spends its share. Same strict-early shape, but now you can peek whenever you need to. That’s what gsDesign’s sfLDOF does, and why everyone just calls it “O’Brien-Fleming” anyway! <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f64c.png" alt="🙌" class="wp-smiley" style="height: 1em; max-height: 1em;" /></p>
<p>Too many words? Yea, I know how you feel. Let’s break things down and code it to understand it.</p>
<p>Let’s start with the simplest O’Brien-Fleming method. We will simulate 3 looks at n=500, with 100,000 trials first and calculate the zscores for each look.</p>
<pre>n &lt;- 500
prob &lt;- 0.5
trial &lt;- 100000
k &lt;- 3
zscore_mat &lt;- matrix(nrow = trial, ncol = k)
gp &lt;- round(n/k)
peek &lt;- c(gp, gp*2, n)

for (trial_i in 1:trial) {
  # for reproducibility
  set.seed(trial_i)
  
  # get sim data
  control &lt;- rbinom(n,1,prob)
  rx &lt;- rbinom(n,1,prob)
  
  # peek at respective k
  for (i in 1:k) {
    m  &lt;- peek[i]
    pA &lt;- mean(rx[1:m])
    pB &lt;- mean(control[1:m])
    p_pool &lt;- (pA + pB) / 2
    se &lt;- sqrt(p_pool * (1 - p_pool) * (2 / m))
    
    # calculate z score, record it
    zscore_mat[trial_i, i] &lt;- abs(pA - pB) / se
  }
}
</pre>



<h3 id="obrien-method">O’Brien-Fleming Method
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#obrien-method" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h3>
<ol>
<li>Start with final z score, let’s choose 1.96 as a guess and search upwards.</li>
<li>Then back calculate second and first look by dividing by square root of information fraction (t). For example, if we have 3 looks, the first look will be 1/3 of the information, second look will be 2/3 of the information and final look will be 3/3 of the information.</li>
<li>Then calculate the total family wise error rates for each trial and call it <code>alpha_final</code>.</li>
<li>Then look at the simulated data and filter out <code>alpha_final</code> between 0.045 and 0.05, then see what z scores for first and second look.</li>
</ol>
<pre># obrien fleming
df &lt;- data.frame(zscore_mat) |&gt;
  as_tibble()

final_z_seq &lt;- seq(1.96, 2.5, 0.001)
t2_z_s_seq &lt;- final_z_seq / sqrt(2/3)
t1_z_s_seq &lt;- final_z_seq / sqrt(1/3)

for (i in 1:length(final_z_seq)) {

# record 1 if zscore passes the respective threshold for each look
alpha_final &lt;- df |&gt;
  mutate(crossed = case_when(
    X3 &gt; final_z_seq[i] ~ 1,
    X2 &gt; t2_z_s_seq[i] ~ 1,
    X1 &gt; t1_z_s_seq[i] ~ 1,
    TRUE ~ 0
  )) |&gt;
  pull(crossed) |&gt;
  mean()

if (alpha_final &lt; 0.05 &#038; alpha_final &gt;= 0.045) { print(paste0(&quot;total alpha: &quot;, alpha_final ,&quot; .found it, it&#39;s on &quot;,i, &quot; &quot;, t1_z_s_seq[i], &quot; &quot;, t2_z_s_seq[i], &quot; &quot;,final_z_seq[i])) }
}

## [1] &quot;total alpha: 0.0494499999999998 .found it, it&#39;s on 65 3.50567083451941 2.47888361969658 2.024&quot;
## [1] &quot;total alpha: 0.0463299999999999 .found it, it&#39;s on 66 3.50740288532698 2.48010836456797 2.025&quot;
## [1] &quot;total alpha: 0.0454099999999999 .found it, it&#39;s on 67 3.50913493613455 2.48133310943936 2.026&quot;
</pre><p>Nice! That worked! Alright, now let’s look at the Lan De Mat & Obrien Fleming method. The idea is to calculate the alpha spent at each look and then find the z scores for each look.</p>




<h3 id="lan">Lan De Mat & Obrien Fleming Method
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#lan" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h3>
<p>So Lan De Mat introduced spend function, which makes the O’Brien-Fleming method more flexible. The idea is to calculate the alpha spent at each look and then find the z scores for each look. The formula for the spend function is <code>2 - 2 * pnorm(qnorm(1 - alpha/2) / sqrt(t))</code>. 
<a href="https://academic.oup.com/biomet/article-abstract/70/3/659/247777" rel="nofollow" target="_blank">here</a> is the paper that derives it, might be a good blog post next time to learn. For now we’ll take it as is and roll with it. The idea is that it’s the chance a random walk crosses a fixed line by time t (the reflection principle)</p>
<ol>
<li>Calculate spending for look 1 with the formula. (note the alpha here is 0.025 for one tail). This spending will be in percentage (like p val).</li>
<li>Sort descending of the first look zscore from our simulation data, then remove the spending percentage of the first look, but record the first zscore value (that would be our first z score on first look). Save into a new dataframe. PS: this is like saying, I’m OK with the false positive of these much on first look.</li>
<li>Calculate the spending for look 2 with the formula, the subtract the first look spending.</li>
<li>Sort descending of the second look zscore of the new dataframe, then record the second zscore value, and remove the spending percentage of the second look on the new dataframe.</li>
<li>Calculate the spending for look 3 (which is 0.025), then subtract of the third look spending to second+one (cumulative) look spending, then do the same as above, record zscore and filter out the rest based on spending 3 percentage.</li>
</ol>
<p>Wow, ok too many words above. Use code to help understand.</p>
<pre># calc spend1 
t &lt;- 1/3                                   
alpha &lt;- 0.025                                 
spend1 &lt;- 2 - 2 * pnorm(qnorm(1 - alpha/2) / sqrt(t))

# filter for those X1 with less than zscore1
df1 &lt;- data.frame(zscore_mat) |&gt;
  as_tibble() |&gt;
  arrange(desc(X1)) |&gt;
  slice(round(spend1*trial*2+1):trial)

zscore1 &lt;- data.frame(zscore_mat) |&gt;
  as_tibble() |&gt;
  arrange(desc(X1)) |&gt;
  slice_head(n = round(spend1*trial*2)) |&gt;
  slice_tail(n = 1) |&gt;
  pull(X1)

# calc spend2
t &lt;- 2/3
spend2_cum &lt;- 2 - 2 * pnorm(qnorm(1 - alpha/2) / sqrt(t))   
spend2 &lt;- spend2_cum - spend1                                

# find how many to look at 
head_num &lt;- round(spend2 * 2 * trial)                                             

# head of however many false positive allowed
zscore2 &lt;- df1 |&gt;
  arrange(desc(X2)) |&gt; 
  slice_head(n = head_num) |&gt;
  slice_tail(n = 1) |&gt;
  pull(X2)

df2 &lt;- df1 |&gt;
  arrange(desc(X2)) |&gt;
  slice((head_num+1):nrow(df1))

# calc spend3
spend3 &lt;- alpha - spend2_cum      
head_num &lt;- round(spend3 * 2 * trial)                

zscore3 &lt;- df2 |&gt;
  arrange(desc(X3)) |&gt;
  slice(head_num) |&gt;
  pull(X3)

zscore1 ; zscore2 ; zscore3

## [1] 3.612396

## [1] 2.480686

## [1] 2.023874
</pre><p>Alright, not too shabby. Looks OK. Let’s use new data to calculate our total family wise error rate.</p>
<pre>### check family wise error
count_false &lt;- vector(mode = &quot;numeric&quot;, length = trial)

for (trial_i in 1:trial) {
  # for reproducibility
  set.seed(trial_i + trial)
  
  # get sim data
  control &lt;- rbinom(n,1,prob)
  rx &lt;- rbinom(n,1,prob)
  
  # peek at respective k
  for (i in 1:length(peek)) {
    m  &lt;- peek[i]
    pA &lt;- mean(rx[1:m])
    pB &lt;- mean(control[1:m])
    p_pool &lt;- (pA + pB) / 2
    se &lt;- sqrt(p_pool * (1 - p_pool) * (2 / m))
    
    # calculate z score
    zscore_vec[i] &lt;- abs(pA - pB) / se
  }
  # new bounds we found 
  bounds &lt;- c(zscore1, zscore2, zscore3) 
  if (any(zscore_vec &gt; bounds)) {count_false[trial_i] &lt;- 1}
}

# what percent of family wise error?
(val &lt;- count_false |&gt; mean())

## [1] 0.05027
</pre><p>Sweet! <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f64c.png" alt="🙌" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Alright, let’s see how the professionals do it?</p>
<pre>## gsDesign
d0 &lt;- gsDesign(k = 3, test.type = 2, alpha = 0.025, sfu = sfLDOF)
d0$upper$bound      

## [1] 3.710303 2.511427 1.993048
</pre>



<h3 id="compare">Comparison
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#compare" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h3>
<p>Let’s put them all on a table</p>
<table>
<thead>
<tr>
<th style="text-align:right">look</th>
<th style="text-align:right">O’Brien-Fleming sim</th>
<th style="text-align:right">O’Brien-Fleming gsDesign</th>
<th style="text-align:right">Lan-DeMets sim</th>
<th style="text-align:right">Lan-DeMets gsDesign</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right">1</td>
<td style="text-align:right">3.51</td>
<td style="text-align:right">3.47</td>
<td style="text-align:right">3.61</td>
<td style="text-align:right">3.71</td>
</tr>
<tr>
<td style="text-align:right">2</td>
<td style="text-align:right">2.48</td>
<td style="text-align:right">2.45</td>
<td style="text-align:right">2.48</td>
<td style="text-align:right">2.51</td>
</tr>
<tr>
<td style="text-align:right">3</td>
<td style="text-align:right">2.02</td>
<td style="text-align:right">2.00</td>
<td style="text-align:right">2.02</td>
<td style="text-align:right">1.99</td>
</tr>
</tbody>
</table>
<p>Not bad at all! <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f64c.png" alt="🙌" class="wp-smiley" style="height: 1em; max-height: 1em;" /> Our sims land within about 0.1 of the professionals on both methods. The small gaps are from our binary outcome ?maybe. The z-scores come in little jumps, so we can’t hit the boundaries exactly. Also, take note that we got here by brute-force simulation, not the real math. gsDesign uses the multivariate normal distribution with the correlation matrix between looks (remember sqrt(t1/t2)?) to calculate these boundaries exactly. That’s a project for next time!</p>




<h2 id="opportunities">Opportunities For Improvement
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#opportunities" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<ul>
<li>Calculate the boundaries exactly with mvtnorm::pmvnorm (no simulation noise)</li>
<li>Simulate power under a real effect, not just the null</li>
<li>Add a futility boundary (test.type = 4) to stop when it’s clearly going nowhere</li>
<li>Look at how stopping early overestimates the effect size</li>
<li>Use a continuous outcome to get rid of the binary lumpiness</li>
<li>Learn derivation of reflection principle behind the Lan-DeMets formula</li>
</ul>




<h2 id="lessons">Lessons learnt
  <a href="https://www.kenkoonwong.com/blog/interim-analysis/#lessons" rel="nofollow" target="_blank"><svg class="anchor-symbol" aria-hidden="true" height="26" width="26" viewBox="0 0 22 22" xmlns="http://www.w3.org/2000/svg">
      <path d="M0 0h24v24H0z" fill="currentColor"></path>
      <path d="M3.9 12c0-1.71 1.39-3.1 3.1-3.1h4V7H7c-2.76.0-5 2.24-5 5s2.24 5 5 5h4v-1.9H7c-1.71.0-3.1-1.39-3.1-3.1zM8 13h8v-2H8v2zm9-6h-4v1.9h4c1.71.0 3.1 1.39 3.1 3.1s-1.39 3.1-3.1 3.1h-4V17h4c2.76.0 5-2.24 5-5s-2.24-5-5-5z"></path>
    </svg></a>
</h2>
<ul>
<li>learnt the simple interim analysses</li>
<li>learnt basic <code>gsDesign</code></li>
<li>learnt that alpha borrowing in interim analyses is not the same as multiplicity, because the second look is correlated to the first look. Bonferroni treats them as if they were barely related, so it over-adjusts. <img src="https://s.w.org/images/core/emoji/13.0.0/72x72/1f914.png" alt="🤔" class="wp-smiley" style="height: 1em; max-height: 1em;" /></li>
<li>learnt interim analyses true zscore estimation requires multivariate normal distribution</li>
</ul>
<p>If you like this article:</p>
<ul>
<li>please feel free to send me a 
<a href="https://www.kenkoonwong.com/blog/" rel="nofollow" target="_blank">comment or visit my other blogs</a></li>
<li>please feel free to follow me on 
<a href="https://bsky.app/profile/kenkoonwong.bsky.social" rel="nofollow" target="_blank">BlueSky</a>, 
<a href="https://twitter.com/kenkoonwong/" rel="nofollow" target="_blank">twitter</a>, 
<a href="https://github.com/kenkoonwong/" rel="nofollow" target="_blank">GitHub</a> or 
<a href="https://rstats.me/@kenkoonwong" rel="nofollow" target="_blank">Mastodon</a></li>
<li>if you would like collaborate please feel free to 
<a href="https://www.kenkoonwong.com/contact/" rel="nofollow" target="_blank">contact me</a></li>
</ul>

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.kenkoonwong.com/blog/interim-analysis/"> r on Everyday Is A School Day</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/interim-analysis-my-messy-notes/">Interim Analysis: My Messy Notes</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404902</post-id>	</item>
		<item>
		<title>pyr.js: Python and R in HTML pages, with one script tag</title>
		<link>https://www.r-bloggers.com/2026/11/pyr-js-python-and-r-in-html-pages-with-one-script-tag/</link>
		
		<dc:creator><![CDATA[T. Moudiki]]></dc:creator>
		<pubDate>Sun, 15 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://thierrymoudiki.github.io//blog/2026/11/15/python/r/javascript/pyr_js</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; "> pyr.js runs Python and R code directly in HTML pages, in the browser, with one script tag: no server, no build step, no install. Pyodide and webR do the heavy lifting.</div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/pyr-js-python-and-r-in-html-pages-with-one-script-tag/">pyr.js: Python and R in HTML pages, with one script tag</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://thierrymoudiki.github.io//blog/2026/11/15/python/r/javascript/pyr_js"> T. Moudiki's Webpage - R</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<p>What if you could put <strong>Python</strong> or <strong>R</strong> code in an HTML page, double-click the file, and have it run? No server, no build step, no install, and nothing sent anywhere: everything runs in the browser.</p>

<p>That’s what <a href="https://github.com/Techtonique/pyr.js" rel="nofollow" target="_blank"><code>pyr.js</code></a> does. I built it this week with Claude (Anthropic’s AI assistant), which wrote the code and tests from my design decisions and verified everything in a headless browser. The heavy lifting is done by <a href="https://pyodide.org/" rel="nofollow" target="_blank">Pyodide</a> (Python compiled to WebAssembly) and <a href="https://docs.r-wasm.org/webr/latest/" rel="nofollow" target="_blank">webR</a> (same thing for R). <code>pyr.js</code> is the glue: a single file, about 460 lines you can actually read, that finds the Python and R code in your page and connects it to your HTML.</p>

<h2 id="a-first-example">A first example</h2>

<p>Save this as an <code>.html</code> file and open it in your browser:</p>

<pre>&lt;script src=&quot;https://cdn.jsdelivr.net/gh/Techtonique/pyr.js@v0.1.0/pyr.js&quot;&gt;&lt;/script&gt;

&lt;p id=&quot;count&quot;&gt;count = 0&lt;/p&gt;
&lt;button py-click=&quot;add_one()&quot; py-out=&quot;count&quot;&gt;Add&lt;/button&gt;

&lt;script type=&quot;text/python&quot;&gt;
count = 0

def add_one():
    global count
    count += 1
    return f&quot;count = {count}&quot;
&lt;/script&gt;
</pre>

<p><code>py-click</code> runs Python when the button is clicked, and <code>py-out</code> says where the result goes. Any DOM event works the same way (<code>py-input</code>, <code>py-change</code>, …), and for R, it’s <code>r-click</code>, <code>r-input</code>, and so on. R’s printed output and plots show up directly in the page:</p>

<pre>&lt;label&gt;Points: &lt;input type=&quot;range&quot; min=&quot;10&quot; max=&quot;500&quot; value=&quot;100&quot;
       r-input=&quot;hist(rnorm(as.numeric(.value)))&quot; r-out=&quot;plot&quot;&gt;&lt;/label&gt;
&lt;div id=&quot;plot&quot;&gt;&lt;/div&gt;
</pre>

<p>Here, <code>.value</code> (<code>_value</code> in Python) holds the slider’s current value, and the histogram is redrawn as you move it.</p>

<h2 id="python-and-r-compute-javascript-draws">Python and R compute, JavaScript draws</h2>

<p>My favorite part: results can be passed as <strong>data</strong> to any JavaScript chart library. In a <a href="https://thierrymoudiki.github.io/blog/2026/09/27/python/r/javascript/analytics-pipeline" rel="nofollow" target="_blank">previous post</a>, I used <a href="https://observablehq.github.io/plot/" rel="nofollow" target="_blank">Observable Plot</a> and <a href="https://www.highcharts.com/" rel="nofollow" target="_blank">Highcharts</a> for dashboards. With <code>pyr.js</code>, <code>py-then</code> / <code>r-then</code> hand the result to a function of your choice. An R data frame arrives as an array of rows, ready to plot:</p>

<pre>&lt;script src=&quot;https://cdn.jsdelivr.net/npm/d3@7.9.2/dist/d3.min.js&quot;&gt;&lt;/script&gt;
&lt;script src=&quot;https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6.19/dist/plot.umd.min.js&quot;&gt;&lt;/script&gt;
&lt;script src=&quot;https://cdn.jsdelivr.net/gh/Techtonique/pyr.js@v0.1.0/pyr.js&quot;&gt;&lt;/script&gt;
&lt;script&gt;
  function draw(rows, el) {
    el.replaceChildren(Plot.dot(rows, { x: &quot;wt&quot;, y: &quot;mpg&quot;, stroke: &quot;cyl&quot; }).plot());
  }
&lt;/script&gt;

&lt;script type=&quot;text/r&quot;&gt;
cars &lt;- function() {
  d &lt;- mtcars
  d$cyl &lt;- factor(paste(d$cyl, &quot;cylinders&quot;))   # factors arrive as text
  d[, c(&quot;wt&quot;, &quot;mpg&quot;, &quot;cyl&quot;)]
}
&lt;/script&gt;

&lt;div r-ready=&quot;cars()&quot; r-then=&quot;draw&quot; r-out=&quot;chart&quot;&gt;&lt;/div&gt;
&lt;div id=&quot;chart&quot;&gt;&lt;/div&gt;
</pre>

<p>The same works with Python: lists of dicts, numpy arrays and pandas DataFrames are converted for you. Here’s the Observable Plot example from the repository, with a Python random walk (redrawn by a slider) and the R data frame above:</p>

<p><img src="https://i2.wp.com/thierrymoudiki.github.io/images/2026-11-15/2026-11-15-image1.png?w=578&#038;ssl=1" alt="image-title-here" class="img-responsive" data-recalc-dims="1" /></p>

<h2 id="how-fast-is-it">How fast is it?</h2>

<p>The first visit downloads the engines (several MB for Python, cached by the browser afterwards), and starting Python takes about 3 seconds. Nothing magical so far.</p>

<p>The interesting part: when <code>pyr.js</code> is hosted on the same site as your pages, Python runs in a <em>shared worker</em>. Other tabs of the site, and even page reloads, reuse the interpreter that’s already running: in my tests (Chromium), the page was ready in <strong>under 0.1 second</strong> instead of 3. Each tab still has its own variables, and a reload still starts the page fresh. For R, webR doesn’t allow this (yet), so it starts with each page.</p>

<p>There’s also a <em>live editing</em> mode for development: put your Python code in a <code>.py</code> file, add <code>data-watch</code> to the script tag, and your changes apply to the running page as soon as you save, without reloading.</p>

<h2 id="privacy">Privacy</h2>

<p>pyr.js sends nothing anywhere. No analytics, no telemetry: your code and your data stay in the visitor’s browser. The only network requests are for the engines themselves and the packages your code imports.</p>

<h2 id="try-it">Try it</h2>

<p>The repository is on <a href="https://github.com/Techtonique/pyr.js" rel="nofollow" target="_blank">GitHub</a> (MIT license), with <strong>four examples</strong>: the basics, <a href="https://observablehq.com/@observablehq/plot-gallery" rel="nofollow" target="_blank">Observable Plot</a>, <a href="https://www.highcharts.com/" rel="nofollow" target="_blank">Highcharts</a> and live editing. The README explains the rest (attributes, data conversion, self-hosting the engines, pinning a version with an <code>integrity</code> hash). Feedback, issues and stars are welcome!</p>

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://thierrymoudiki.github.io//blog/2026/11/15/python/r/javascript/pyr_js"> T. Moudiki's Webpage - R</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/pyr-js-python-and-r-in-html-pages-with-one-script-tag/">pyr.js: Python and R in HTML pages, with one script tag</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404899</post-id>	</item>
		<item>
		<title>Introducing PLaMo-based AI translation for Bioconductor lessons</title>
		<link>https://www.r-bloggers.com/2026/11/introducing-plamo-based-ai-translation-for-bioconductor-lessons/</link>
		
		<dc:creator><![CDATA[Kozo Nishida]]></dc:creator>
		<pubDate>Fri, 13 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://blog.bioconductor.org/posts/2026-11-13-ai-translation/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>In the report on our Japanese workshops, I explained why our Crowdin integration had become difficult to maintain. Keeping executable R Markdown lessons in sync required file conversions, translation synchronisation and reconstruction of buildab...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/introducing-plamo-based-ai-translation-for-bioconductor-lessons/">Introducing PLaMo-based AI translation for Bioconductor lessons</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://blog.bioconductor.org/posts/2026-11-13-ai-translation/"> Bioconductor community blog</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 





<p>In the <a href="https://blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/" rel="nofollow" target="_blank">report on our Japanese workshops</a>, I explained why our Crowdin integration had become difficult to maintain. Keeping executable R Markdown lessons in sync required file conversions, translation synchronisation and reconstruction of buildable lessons. I remain grateful for Crowdin’s support. We are now moving to PLaMo-based AI translation directly in the lesson repositories, with code and markup protected and community review on GitHub.</p>
<p>I am introducing a new AI-assisted translation workflow in the <a href="https://github.com/bioconductor-translations" rel="nofollow" target="_blank">bioconductor-translations</a> organisation, built around PLaMo 3 Translate and review on GitHub. The first lesson is <a href="https://carpentries-incubator.github.io/bioc-intro/" rel="nofollow" target="_blank"><em>Introduction to data analysis with R and Bioconductor</em></a>, with published editions in Japanese, Spanish, Simplified Chinese, French and Urdu. These editions are open for community review and improvement.</p>
<section id="what-is-plamo-3-translate" class="level2">
<h2 class="anchored" data-anchor-id="what-is-plamo-3-translate">What is PLaMo 3 Translate?</h2>
<p>PLaMo 3 Translate is a family of large language models developed by Preferred Networks (PFN) specifically for translation. PFN’s November 5, 2026 technical article, <a href="https://www.preferred.jp/en/blog/tech/plamo-3-translate-weight-release" rel="nofollow" target="_blank"><em>PLaMo 3 Translate: World-Class Translation with Open Weights—Here’s How We Built It</em></a>, explains the development process and the release of downloadable model weights.</p>
<p>PFN trains the models on source–translation pairs and further refines them to prefer better translations. Smaller models also learn from the 31B model through distillation. The family supports 53 languages and can take surrounding context and a glossary into account. These features fit our lesson workflow: we can supply nearby explanations and preferred technical terminology when translating a passage.</p>
<p>The models come in four sizes: 0.8B, 2B, 8B and 31B parameters. Our workflow uses <a href="https://huggingface.co/pfnet/plamo-3-translate-8b-MLX-4bit" rel="nofollow" target="_blank"><code>pfnet/plamo-3-translate-8b-MLX-4bit</code></a>, the 8B model in a 4-bit MLX format that reduces memory use for local generation on Apple silicon Macs. The model weights are distributed under the PLaMo Community License.</p>
</section>
<section id="explore-the-five-language-editions" class="level2">
<h2 class="anchored" data-anchor-id="explore-the-five-language-editions">Explore the five language editions</h2>
<p>The introductory lesson covers organising data in spreadsheets, using R and RStudio, manipulating data with <code>dplyr</code>, and visualising data. Each language edition has a lesson website for learners and a repository for contributors:</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th style="text-align: left;">Language</th>
<th style="text-align: left;">Read the lesson</th>
<th style="text-align: left;">Contribute on GitHub</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Japanese</td>
<td style="text-align: left;"><a href="https://bioconductor-translations.github.io/bioc-intro-plamo3translate-ja/" rel="nofollow" target="_blank">Japanese lesson</a></td>
<td style="text-align: left;"><a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-ja" rel="nofollow" target="_blank">bioc-intro-plamo3translate-ja</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">Spanish</td>
<td style="text-align: left;"><a href="https://bioconductor-translations.github.io/bioc-intro-plamo3translate-es/" rel="nofollow" target="_blank">Spanish lesson</a></td>
<td style="text-align: left;"><a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-es" rel="nofollow" target="_blank">bioc-intro-plamo3translate-es</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">Simplified Chinese</td>
<td style="text-align: left;"><a href="https://bioconductor-translations.github.io/bioc-intro-plamo3translate-zh-hans/" rel="nofollow" target="_blank">Simplified Chinese lesson</a></td>
<td style="text-align: left;"><a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-zh-hans" rel="nofollow" target="_blank">bioc-intro-plamo3translate-zh-hans</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">French</td>
<td style="text-align: left;"><a href="https://bioconductor-translations.github.io/bioc-intro-plamo3translate-fr/" rel="nofollow" target="_blank">French lesson</a></td>
<td style="text-align: left;"><a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-fr" rel="nofollow" target="_blank">bioc-intro-plamo3translate-fr</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">Urdu</td>
<td style="text-align: left;"><a href="https://bioconductor-translations.github.io/bioc-intro-plamo3translate-ur/" rel="nofollow" target="_blank">Urdu lesson</a></td>
<td style="text-align: left;"><a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-ur" rel="nofollow" target="_blank">bioc-intro-plamo3translate-ur</a></td>
</tr>
</tbody>
</table>
<p>Our Crowdin project also received Portuguese translation contributions, and I am grateful to everyone who contributed them. I am sorry that Portuguese is not included among these editions: the PLaMo translation model we currently use does not support Portuguese, which is absent from its <a href="https://huggingface.co/pfnet/plamo-3-translate-8b#supported-languages" rel="nofollow" target="_blank">supported-language list</a>.</p>
<p>Translation coverage and review are still in progress. Some setup pages and interface labels remain in English. At the time of writing, the published Urdu episodes are available, but that repository’s <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-ur/blob/main/translation/config.json" rel="nofollow" target="_blank">updater configuration</a> and translation guide still refer to Japanese and need adaptation before being used to regenerate Urdu text.</p>
</section>
<section id="how-the-translation-system-works" class="level2">
<h2 class="anchored" data-anchor-id="how-the-translation-system-works">How the translation system works</h2>
<div id="fig-translation-workflow" class="quarto-float quarto-figure quarto-figure-center anchored" alt="Workflow diagram: English lesson prose either reuses matching translation memory or is translated locally by PLaMo using a glossary and context. Protected code and lesson markup bypass generation. The reassembled lessons undergo structure checks, community review and a build. Reviewed corrections are saved to translation memory." data-fig-align="center">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-translation-workflow-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://i0.wp.com/blog.bioconductor.org/posts/2026-11-13-ai-translation/translation-workflow.jpg?w=450&#038;ssl=1" class="img-fluid quarto-figure quarto-figure-center figure-img" alt="Workflow diagram: English lesson prose either reuses matching translation memory or is translated locally by PLaMo using a glossary and context. Protected code and lesson markup bypass generation. The reassembled lessons undergo structure checks, community review and a build. Reviewed corrections are saved to translation memory."  data-recalc-dims="1">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-translation-workflow-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 1: Matching passages reuse translation memory; new or changed prose is translated with PLaMo. Protected code and markup rejoin the text before structural checks, community review and a lesson build.
</figcaption>
</figure>
</div>
<ol type="1">
<li><strong>Read an English source revision.</strong> The translator reads the upstream lesson’s R Markdown files from a Git revision without checking it out over the translated working files.</li>
<li><strong>Reuse matching translations.</strong> Each translation repository stores its versioned translation memory in <code>translation/seed-cache.json</code>. This JSON file stores previous translations and review statuses by source passage and episode. Matching records can be reused, while new or changed prose is sent to the model.</li>
<li><strong>Guide terminology and protect structure.</strong> A language-specific glossary and contextual guidance help disambiguate terms. Code blocks, inline code, link destinations and lesson markup are protected during translation, and structural checks run before lesson files are replaced.</li>
<li><strong>Review and build.</strong> Maintainers inspect the translated text and memory changes, build the lesson, and commit the reviewed result. The translator itself does not commit or push changes.</li>
</ol>
<section id="shared-terminology-in-glossary.json" class="level3">
<h3 class="anchored" data-anchor-id="shared-terminology-in-glossary.json">Shared terminology in glossary.json</h3>
<p>Each repository keeps its preferred terminology in <code>translation/glossary.json</code>. For example, the Japanese repository’s <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-ja/blob/main/translation/glossary.json" rel="nofollow" target="_blank">translation/glossary.json</a> contains <code>source</code> and <code>target</code> term pairs, mapping “data frame” to “データフレーム” and keeping “Bioconductor” unchanged. The translator passes these pairs to the model to guide terminology consistently across episodes. Contributors can propose glossary changes through pull requests. A glossary change requires fresh generation for machine records on the next update; matching <code>reviewed</code> records retain their wording, so terminology corrections must also be applied to the reviewed translation memory where needed.</p>
</section>
<section id="update-a-translation-yourself" class="level3">
<h3 class="anchored" data-anchor-id="update-a-translation-yourself">Update a translation yourself</h3>
<p>To update a translation yourself, follow <code>translation/README.md</code> in the repository for your language. These documents cover environment setup, fetching the English source, previewing changes, generating translations, reviewing the results and building the lesson:</p>
<ul>
<li>Japanese: <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-ja/blob/main/translation/README.md" rel="nofollow" target="_blank">translation/README.md</a></li>
<li>Spanish: <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-es/blob/main/translation/README.md" rel="nofollow" target="_blank">translation/README.md</a></li>
<li>Simplified Chinese: <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-zh-hans/blob/main/translation/README.md" rel="nofollow" target="_blank">translation/README.md</a></li>
<li>French: <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-fr/blob/main/translation/README.md" rel="nofollow" target="_blank">translation/README.md</a></li>
</ul>
<p>For Urdu, the updater configuration and documentation caveat above still applies.</p>
<p>For example, in the Japanese repository, after configuring the upstream remote as described in its guide, a maintainer can run:</p>
<pre>git fetch upstream main
uv run --locked scripts/translate.py --source-ref upstream/main --dry-run
uv run --locked scripts/translate.py --source-ref upstream/main --update-memory
git diff -- episodes translation
Rscript scripts/build.R</pre>
<p>The dry run shows which passages can be reused and which need translation, without generating new text or writing lesson files. The <code>--update-memory</code> option saves new translation records to <code>translation/seed-cache.json</code> and source provenance to <code>translation/source-manifest.json</code> for review. The local runtime cache under <code>.cache/translation/</code> is ignored by Git. Other editions may use a different remote name, so follow <code>translation/README.md</code> in the repository you are updating.</p>
<p>The translator also checks for local edits and stops before overwriting an edited lesson. Small build adaptations are recorded separately with checks against the expected source. Images, datasets and episode ordering still need separate maintenance when the English lesson changes.</p>
</section>
</section>
<section id="local-generation-reusable-community-knowledge" class="level2">
<h2 class="anchored" data-anchor-id="local-generation-reusable-community-knowledge">Local generation, reusable community knowledge</h2>
<p>The workflow pins the model revision and locks its Python dependencies, recording the source revision and model provenance with the translation resources.</p>
<p>Generating new translations with the current implementation requires an Apple silicon Mac, Python 3.12 and <code>uv</code>; the first model download is about 6 GB. Planning updates and replaying cached translations can run with Python 3.12 alone on other platforms. Reading lessons and contributing a review on GitHub require no model installation.</p>
<p>The translation memory gives community review a lasting role. You can inspect the records in the Japanese translation repository’s <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-ja/blob/main/translation/seed-cache.json" rel="nofollow" target="_blank">translation/seed-cache.json</a>. Newly generated records are marked <code>machine</code>. After checking a record in <code>translation/seed-cache.json</code>, a reviewer can mark it <code>reviewed</code> so that its wording is retained for matching source text in the same episode and language pair, even when model or glossary settings change. The French translation repository’s <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-fr/blob/main/translation/README.md" rel="nofollow" target="_blank">translation/README.md</a> explains this review process.</p>
</section>
<section id="help-make-the-lessons-better" class="level2">
<h2 class="anchored" data-anchor-id="help-make-the-lessons-better">Help make the lessons better</h2>
<p>A successful build shows that the lesson can be rendered, but readers and instructors still need to check its meaning. A translation can sound fluent while changing an exercise condition, using an unfamiliar technical term, or losing an important distinction. The Simplified Chinese translation repository’s <a href="https://github.com/bioconductor-translations/bioc-intro-plamo3translate-zh-hans/blob/main/translation/README.md" rel="nofollow" target="_blank">translation/README.md</a> explicitly identifies its initial translation memory as machine generated and needing review.</p>
<p>If you use one of these languages, please try an episode and compare it with the English lesson. We especially welcome help with:</p>
<ul>
<li><strong>Language and terminology:</strong> suggest clearer explanations and vocabulary familiar to your local research community.</li>
<li><strong>Exercises and teaching:</strong> check that questions, solutions and instructor notes preserve the original learning goals.</li>
<li><strong>Corrections that carry forward:</strong> help maintainers update the glossary, context guidance and translation memory alongside the lesson text.</li>
</ul>
<p>Open an issue in the relevant repository with the episode name, the passage you are referring to, and a suggested correction or explanation. Pull requests are welcome too; please follow that repository’s contribution and translation guides. If you are interested in reviewing translations, adding support for another language, or translating another lesson, join the discussion in the #documentation-translation channel on the <a href="https://chat.bioconductor.org/" rel="nofollow" target="_blank">Bioconductor community chat</a>.</p>
<p>Thank you to the original lesson authors and maintainers, and to everyone translating, reviewing and teaching these materials. This work builds on the community’s translation efforts described in the Japanese workshop report and is supported by the <a href="https://bioconductor.org/developers/developer-champions-program/" rel="nofollow" target="_blank">Bioconductor Developer Champions Program</a>.</p>


</section>

<p>
© 2026 Bioconductor. Content is published under <a href="https://creativecommons.org/licenses/by/4.0/" rel="nofollow" target="_blank">Creative Commons CC-BY-4.0 License</a> for the text and <a href="https://opensource.org/licenses/BSD-3-Clause" rel="nofollow" target="_blank">BSD 3-Clause License</a> for any code. | <a href="https://www.r-bloggers.com/" rel="nofollow" target="_blank">R-Bloggers</a>
</p> 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.bioconductor.org/posts/2026-11-13-ai-translation/"> Bioconductor community blog</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/introducing-plamo-based-ai-translation-for-bioconductor-lessons/">Introducing PLaMo-based AI translation for Bioconductor lessons</a>]]></content:encoded>
					
		
		<enclosure url="https://blog.bioconductor.org/posts/2026-11-13-ai-translation/translation-workflow.jpg" length="0" type="image/jpeg" />

		<post-id xmlns="com-wordpress:feed-additions:1">404847</post-id>	</item>
		<item>
		<title>Semantic segmentation in rplatypus and pyplatypus: what is in the box</title>
		<link>https://www.r-bloggers.com/2026/11/semantic-segmentation-in-rplatypus-and-pyplatypus-what-is-in-the-box/</link>
		
		<dc:creator><![CDATA[Michał Maj]]></dc:creator>
		<pubDate>Thu, 12 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://datascienceguts.com/posts/2026-11-12-segmentation-tour/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>The last post took one published model and one dataset and asked where the model is weakest. This one steps back and opens the box: everything rplatypus and pyplatypus offer for semantic segmentation - a mask per class, scored by overlap - and ...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/semantic-segmentation-in-rplatypus-and-pyplatypus-what-is-in-the-box/">Semantic segmentation in rplatypus and pyplatypus: what is in the box</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/"> Data Science Guts</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 






<p>The <a href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">last post</a> took one published model and one dataset and asked where the model is weakest. This one steps back and opens the box: everything rplatypus and pyplatypus offer for semantic segmentation &#8211; a mask per class, scored by overlap &#8211; and how to choose among it all.</p>
<p>Training is what the package is for, and almost every field below is about training a model. To keep the post runnable on any laptop, its code only reads configurations and builds the networks, on the four nuclei images from the <a href="https://datascienceguts.com/posts/2026-11-03-rplatypus/" rel="nofollow" target="_blank">first post</a>; the scores in it come from full training runs of the same configurations.</p>
<section id="one-file-holds-the-experiment" class="level2">
<h2 class="anchored" data-anchor-id="one-file-holds-the-experiment">One file holds the experiment</h2>
<p>Everything below can be passed to constructors in R or Python, but the clearest way to see it is the file both packages read. This one describes a whole training run:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre>nuclei.yaml</pre>
</div>
<pre>task: semantic_segmentation

data:
  train_path: nuclei
  validation_path: nuclei
  mode: nested_dirs
  colormap:
    - [0, 0, 0]          # background
    - [255, 255, 255]    # nucleus

models:
  - name: unet
    architecture: u_net
    input_shape: [256, 256]
    blocks: 4
    filters: 16
    loss:
      name: cce_dice
    metrics:
      - name: dice
        include_background: false
    optimizer:
      name: adam
      learning_rate: 0.001
    callbacks:
      - name: early_stopping
        monitor: val_dice
        patience: 5
    augmentation:
      - name: HorizontalFlip
        params: {p: 0.5}
      - name: RandomRotate90
        params: {p: 0.5}
    epochs: 30
    batch_size: 8</pre>
</div>
<p>Read from the top:</p>
<ul>
<li><strong><code>task</code></strong> decides what the rest of the file may contain. Semantic segmentation is one of four; the others &#8211; classification, instance segmentation and object detection &#8211; get posts of their own.</li>
<li><strong><code>data</code></strong> says where the images are and how to read them:
<ul>
<li><code>train_path</code> and <code>validation_path</code> say where the training and validation data are;</li>
<li><code>mode</code> is how the files are arranged: <code>nested_dirs</code> is one directory per image with <code>images/</code> and <code>masks/</code> inside, <code>config_file</code> is a CSV listing the files, which is what <code>split_dataset()</code> writes;</li>
<li><strong><code>colormap</code></strong> says which colour in a mask means which class, one entry per class, background first.</li>
</ul></li>
<li><strong><code>models</code></strong> is a list, so one file can train several models on the same data and compare them. Each entry has:
<ul>
<li>a <code>name</code>, an <code>architecture</code> and an <code>input_shape</code> &#8211; these are required;</li>
<li><code>blocks</code>, the depth;</li>
<li><code>filters</code>, the width of the first block, doubling at each level down.</li>
</ul></li>
<li><strong><code>weights</code></strong> and <strong><code>fit</code></strong> are not in this file, because it trains from scratch:
<ul>
<li><code>weights</code> names a checkpoint &#8211; a published name from <code>available_weights()</code>, a Hugging Face reference or a local file &#8211; and training then starts from it rather than from random numbers;</li>
<li><code>fit: false</code> means no training at all: the weights are loaded and the model is ready for <code>predict()</code> and <code>evaluate()</code>, which is how the <a href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">last post</a> scored the published nuclei model.</li>
</ul></li>
<li><strong><code>loss</code></strong>, <strong><code>metrics</code></strong>, <strong><code>optimizer</code></strong>, <strong><code>callbacks</code></strong> and <strong><code>augmentation</code></strong> each take a name and that component’s own arguments. Every name comes from a catalogue the package can print, which is most of the rest of this post.</li>
</ul>
<p>Reading it gives the same object as building it from arguments:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-1-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-1-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<div class="cell">
<pre>library(rplatypus)

spec &lt;- platypus_spec(&quot;nuclei.yaml&quot;)
spec
#&gt; platypus specification
#&gt;   from        : nuclei.yaml 
#&gt;   data        : nuclei 
#&gt;   classes     : 2 
#&gt;   models      : 1 
#&gt;     unet           u_net            256x256, cce_dice, 30 epochs</pre>
</div>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<pre>import pyplatypus as pp

spec = pp.from_yaml(&quot;nuclei.yaml&quot;)
[(m.name, m.architecture.value, m.input_shape) for m in spec.models]
#&gt; [('unet', 'u_net', (256, 256))]</pre>
</div>
</div>
</div>
<p>The file is checked before a single image is read or the network is built, so a mistake costs a second rather than an epoch. Asking for something the architecture does not have is refused with the field that is wrong:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-2-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-2-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-2-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-2-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-2-1" class="tab-pane active" aria-labelledby="tabset-2-1-tab">
<div class="cell">
<pre>platypus_spec(
  data = segmentation_data(&quot;nuclei&quot;, &quot;nuclei&quot;, colormap = binary_colormap),
  models = list(u_net(&quot;x&quot;, input_shape = c(256, 256),
                      deep_supervision = TRUE))
)
#&gt; Error in `platypus_spec()`:
#&gt; ! The specification has a problem:
#&gt;   * models[[1]]: deep_supervision requires architecture 'u_net_plus_plus'; 'u_net' has no intermediate outputs to supervise</pre>
</div>
</div>
<div id="tabset-2-2" class="tab-pane" aria-labelledby="tabset-2-2-tab">
<pre>pp.from_dict({
    &quot;task&quot;: &quot;semantic_segmentation&quot;,
    &quot;data&quot;: {&quot;train_path&quot;: &quot;nuclei&quot;, &quot;validation_path&quot;: &quot;nuclei&quot;,
             &quot;colormap&quot;: [[0, 0, 0], [255, 255, 255]]},
    &quot;models&quot;: [{&quot;name&quot;: &quot;x&quot;, &quot;architecture&quot;: &quot;u_net&quot;,
                &quot;input_shape&quot;: [256, 256], &quot;deep_supervision&quot;: True}],
})
#&gt; ConfigError: The configuration has 1 problem:
#&gt;   - models[0]: deep_supervision requires architecture 'u_net_plus_plus';
#&gt;     'u_net' has no intermediate outputs to supervise</pre>
</div>
</div>
</div>
</section>
<section id="four-architectures" class="level2">
<h2 class="anchored" data-anchor-id="four-architectures">Four architectures</h2>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-3-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-3-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-3-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-3-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-3-1" class="tab-pane active" aria-labelledby="tabset-3-1-tab">
<div class="cell">
<pre>available_models(task = &quot;semantic_segmentation&quot;)$name
#&gt; [1] &quot;u_net&quot;           &quot;u_net_plus_plus&quot; &quot;res_u_net&quot;       &quot;linknet&quot;</pre>
</div>
</div>
<div id="tabset-3-2" class="tab-pane" aria-labelledby="tabset-3-2-tab">
<pre>[m[&quot;name&quot;] for m in pp.available_models(&quot;semantic_segmentation&quot;)]
#&gt; ['u_net', 'u_net_plus_plus', 'res_u_net', 'linknet']</pre>
</div>
</div>
</div>
<p>All four are the same U: an encoder that halves the picture and doubles the features <code>blocks</code> times, and a decoder that climbs back up, with skip connections handing each level’s detail across. They differ in what is inside a block and in how the skips arrive.</p>
<p>Blue blocks are the encoder, green the decoder; each arrow is a connection. The picture going in is a stained tissue image from the same Data Science Bowl collection, and the one coming out is its true mask.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-4-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-4-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">U-Net</a></li><li class="nav-item"><a class="nav-link" id="tabset-4-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-4-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">U-Net++</a></li><li class="nav-item"><a class="nav-link" id="tabset-4-3-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-4-3" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Res-U-Net</a></li><li class="nav-item"><a class="nav-link" id="tabset-4-4-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-4-4" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">LinkNet</a></li></ul>
<div class="tab-content">
<div id="tabset-4-1" class="tab-pane active" aria-labelledby="tabset-4-1-tab">
<div class="cell" data-renderings="["light","dark"]">
<div class="cell-output-display light-content">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i0.wp.com/datascienceguts.com/posts/2026-11-12-segmentation-tour/index_files/figure-html/u-net-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>U-Net: each skip connection goes straight into the decoder block at the same level.</figcaption>
</figure>
</div>
</div>
</div>
<p>The plain U. Each skip connection is <em>concatenated</em> onto the decoder’s features, so a decoder block sees both what came up from below and what the encoder saw at its own scale. It is the one to start with.</p>
</div>
<div id="tabset-4-2" class="tab-pane" aria-labelledby="tabset-4-2-tab">
<div class="cell" data-renderings="["light","dark"]">
<div class="cell-output-display light-content">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/datascienceguts.com/posts/2026-11-12-segmentation-tour/index_files/figure-html/u-net-plus-plus-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>U-Net++: the space between encoder and decoder is filled with nested blocks, each fed by every earlier block in its row and by the one diagonally below.</figcaption>
</figure>
</div>
</div>
</div>
<p>The space between encoder and decoder is filled with nested blocks: each takes every earlier block in its row and the one diagonally below, so the top row &#8211; which is the decoder &#8211; meets the encoder at every depth rather than one. It is the only one of the four that can take <code>deep_supervision</code>, a loss on each block of the top row, which is the refusal above.</p>
</div>
<div id="tabset-4-3" class="tab-pane" aria-labelledby="tabset-4-3-tab">
<div class="cell" data-renderings="["light","dark"]">
<div class="cell-output-display light-content">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/datascienceguts.com/posts/2026-11-12-segmentation-tour/index_files/figure-html/res-u-net-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>Res-U-Net: the U-Net layout with residual blocks &#8211; two layers and a shortcut over them, added back at the end.</figcaption>
</figure>
</div>
</div>
</div>
<p>The same U as U-Net, with residual blocks: each block’s input skips over its two layers and is <em>added</em> back at the end, so the layers learn a correction rather than a whole new representation. The usual reason to try it is depth, when more <code>blocks</code> stop helping.</p>
</div>
<div id="tabset-4-4" class="tab-pane" aria-labelledby="tabset-4-4-tab">
<div class="cell" data-renderings="["light","dark"]">
<div class="cell-output-display light-content">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/datascienceguts.com/posts/2026-11-12-segmentation-tour/index_files/figure-html/linknet-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>LinkNet: residual encoder blocks, skip connections added rather than concatenated, and a lighter decoder.</figcaption>
</figure>
</div>
</div>
</div>
<p>Residual blocks in the encoder, and skip connections that are <em>added</em> to the decoder’s features instead of concatenated onto them. Fewer channels reach each decoder block, so the decoder is lighter. Its argument is memory and speed, not score.</p>
</div>
</div>
</div>
<p>How much each costs at the same depth and width is quick to see. The cheapest way to build all four is one epoch on four small images:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-5-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-5-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-5-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-5-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-5-1" class="tab-pane active" aria-labelledby="tabset-5-1-tab">
<div class="cell">
<pre>one_epoch &lt;- function(architecture, name) {
  architecture(name, input_shape = c(64, 64), blocks = 4, filters = 16,
               epochs = 1, batch_size = 2)
}
four &lt;- platypus_spec(
  data = segmentation_data(&quot;nuclei&quot;, &quot;nuclei&quot;, colormap = binary_colormap),
  models = list(one_epoch(u_net, &quot;u_net&quot;),
                one_epoch(u_net_plus_plus, &quot;u_net_plus_plus&quot;),
                one_epoch(res_u_net, &quot;res_u_net&quot;),
                one_epoch(linknet, &quot;linknet&quot;))
)
built &lt;- platypus_fit(four, verbose = FALSE)
evaluate(built)[, c(&quot;model&quot;, &quot;parameters&quot;)]
#&gt;             model parameters
#&gt; 1           u_net    1942594
#&gt; 2 u_net_plus_plus    2263730
#&gt; 3       res_u_net    2029682
#&gt; 4         linknet    1746754</pre>
</div>
</div>
<div id="tabset-5-2" class="tab-pane" aria-labelledby="tabset-5-2-tab">
<pre>def one_epoch(architecture):
    return {&quot;name&quot;: architecture, &quot;architecture&quot;: architecture,
            &quot;input_shape&quot;: [64, 64], &quot;blocks&quot;: 4, &quot;filters&quot;: 16,
            &quot;epochs&quot;: 1, &quot;batch_size&quot;: 2}

engine = pp.build_engine(pp.from_dict({
    &quot;task&quot;: &quot;semantic_segmentation&quot;,
    &quot;data&quot;: {&quot;train_path&quot;: &quot;nuclei&quot;, &quot;validation_path&quot;: &quot;nuclei&quot;,
             &quot;colormap&quot;: [[0, 0, 0], [255, 255, 255]]},
    &quot;models&quot;: [one_epoch(a) for a in
               [&quot;u_net&quot;, &quot;u_net_plus_plus&quot;, &quot;res_u_net&quot;, &quot;linknet&quot;]],
}))
engine.fit()
[(row[&quot;model&quot;], row[&quot;parameters&quot;]) for row in engine.evaluate()]
#&gt; [('u_net', 1942594), ('u_net_plus_plus', 2263730),
#&gt;  ('res_u_net', 2029682), ('linknet', 1746754)]</pre>
</div>
</div>
</div>
<p>LinkNet is 10% smaller than U-Net here, not half: at 16 filters and four blocks the shared skeleton dominates, and the decoder savings are a small part of it.</p>
<section id="which-one-wins" class="level3">
<h3 class="anchored" data-anchor-id="which-one-wins">Which one wins</h3>
<p>When the nuclei weights were published, all four were trained on the same split of the 2018 Data Science Bowl images, with the same specification, for 60 epochs:</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>architecture</th>
<th style="text-align: right;">parameters</th>
<th style="text-align: right;">mean Dice</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>U-Net</td>
<td style="text-align: right;">1,942,594</td>
<td style="text-align: right;">0.9217</td>
</tr>
<tr class="even">
<td>U-Net++</td>
<td style="text-align: right;">2,263,730</td>
<td style="text-align: right;">0.9212</td>
</tr>
<tr class="odd">
<td>LinkNet</td>
<td style="text-align: right;">1,746,754</td>
<td style="text-align: right;">0.9203</td>
</tr>
<tr class="even">
<td>Res-U-Net</td>
<td style="text-align: right;">2,029,682</td>
<td style="text-align: right;">0.9187</td>
</tr>
</tbody>
</table>
<p>The four means are <strong>0.0030</strong> apart. Within any one of them, the images it was scored on are <strong>0.054</strong> apart. And training the same U-Net again with a different seed moved its mean by 0.0012 &#8211; 40% of the whole gap between architectures.</p>
<p>On this problem the architecture is not where the result comes from, and that is why only one set of weights was published rather than four. It will not be true of every dataset, but it is the reason to start with a plain U-Net and look for a gain somewhere else first &#8211; the data, the loss, the resolution &#8211; before swapping the network.</p>
</section>
</section>
<section id="losses-and-metrics" class="level2">
<h2 class="anchored" data-anchor-id="losses-and-metrics">Losses and metrics</h2>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-6-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-6-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-6-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-6-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-6-1" class="tab-pane active" aria-labelledby="tabset-6-1-tab">
<div class="cell">
<pre>available_losses(task = &quot;semantic_segmentation&quot;)$name
#&gt;  [1] &quot;boundary&quot;         &quot;cce&quot;              &quot;cce_dice&quot;        
#&gt;  [4] &quot;combo&quot;            &quot;dice&quot;             &quot;focal&quot;           
#&gt;  [7] &quot;focal_tversky&quot;    &quot;generalized_dice&quot; &quot;iou&quot;             
#&gt; [10] &quot;lovasz&quot;           &quot;tversky&quot;
available_metrics(task = &quot;semantic_segmentation&quot;)$name
#&gt; [1] &quot;cldice&quot;       &quot;dice&quot;         &quot;iou&quot;          &quot;surface_dice&quot;
#&gt; [5] &quot;tversky&quot;</pre>
</div>
</div>
<div id="tabset-6-2" class="tab-pane" aria-labelledby="tabset-6-2-tab">
<pre>[l[&quot;name&quot;] for l in pp.available_losses(&quot;semantic_segmentation&quot;)]
#&gt; ['boundary', 'cce', 'cce_dice', 'combo', 'dice', 'focal', 'focal_tversky',
#&gt;  'generalized_dice', 'iou', 'lovasz', 'tversky']
[m[&quot;name&quot;] for m in pp.available_metrics(&quot;semantic_segmentation&quot;)]
#&gt; ['cldice', 'dice', 'iou', 'surface_dice', 'tversky']</pre>
</div>
</div>
</div>
<p>The <code>task</code> matters here: classification, detection and instance segmentation each have their own architectures, losses and metrics, and the same functions list them. Each entry also comes with its arguments and a one-line description. The short version of the losses:</p>
<ul>
<li><strong><code>cce_dice</code></strong> &#8211; cross-entropy blended with Dice &#8211; is the one to start a run with.</li>
<li><strong><code>tversky</code></strong> and <strong><code>focal_tversky</code></strong> take an <code>alpha</code> that weighs a missed pixel against a false alarm. Above 0.5 they would rather over-segment than miss, which suits a small class you cannot afford to lose.</li>
<li><strong><code>generalized_dice</code></strong> weights each class by the inverse square of its area, so with several classes of very different size the small ones still count.</li>
<li><strong><code>boundary</code></strong> adds a term for how far a wrong pixel is from the true edge, and <strong><code>lovasz</code></strong> optimises the IoU you are going to report rather than a stand-in for it.</li>
</ul>
<p>The metrics are fewer and each answers a different question. <strong><code>dice</code></strong> is the one to report unless there is a reason not to &#8211; with <code>include_background = FALSE</code>, as the last post explained. <strong><code>cldice</code></strong> asks whether a thin structure stays connected, which a vessel post later in the series needs, and <strong><code>surface_dice</code></strong> asks how much of a boundary lies within a tolerance of the truth, which matters for volumes.</p>
</section>
<section id="training-around-the-model" class="level2">
<h2 class="anchored" data-anchor-id="training-around-the-model">Training, around the model</h2>
<p>The rest of the file is the training run itself, and each part has its catalogue too:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-7-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-7-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-7-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-7-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-7-1" class="tab-pane active" aria-labelledby="tabset-7-1-tab">
<div class="cell">
<pre>available_optimizers()$name
#&gt; [1] &quot;adadelta&quot; &quot;adagrad&quot;  &quot;adam&quot;     &quot;adamax&quot;   &quot;adamw&quot;    &quot;nadam&quot;   
#&gt; [7] &quot;rmsprop&quot;  &quot;sgd&quot;
available_callbacks()$name
#&gt;  [1] &quot;cosine_annealing&quot;      &quot;csv_logger&quot;            &quot;early_stopping&quot;       
#&gt;  [4] &quot;ema&quot;                   &quot;linear_annealing&quot;      &quot;model_checkpoint&quot;     
#&gt;  [7] &quot;reduce_lr_on_plateau&quot;  &quot;step_decay&quot;            &quot;stop_after&quot;           
#&gt; [10] &quot;swa&quot;                   &quot;terminate_on_collapse&quot; &quot;terminate_on_nan&quot;
length(available_transforms())
#&gt; [1] 118</pre>
</div>
</div>
<div id="tabset-7-2" class="tab-pane" aria-labelledby="tabset-7-2-tab">
<pre>[o[&quot;name&quot;] for o in pp.available_optimizers()]
#&gt; ['adadelta', 'adagrad', 'adam', 'adamax', 'adamw', 'nadam', 'rmsprop',
#&gt;  'sgd']
[c[&quot;name&quot;] for c in pp.available_callbacks()]
#&gt; ['cosine_annealing', 'csv_logger', 'early_stopping', 'ema',
#&gt;  'linear_annealing', 'model_checkpoint', 'reduce_lr_on_plateau',
#&gt;  'step_decay', 'stop_after', 'swa', 'terminate_on_collapse',
#&gt;  'terminate_on_nan']
len(pp.available_transforms())
#&gt; 118</pre>
</div>
</div>
</div>
<p>Eight optimisers; twelve callbacks, from early stopping and learning-rate schedules to weight averaging (<code>ema</code>, <code>swa</code>) and two that stop a run that has gone wrong rather than letting it finish; and 118 augmentations from <a href="https://albumentations.ai/" rel="nofollow" target="_blank">albumentations</a>, each named exactly as albumentations names it, with its own parameters under <code>params</code>.</p>
<p>Fifteen of them, on the image from the architecture diagrams, each applied every time (<code>p = 1</code>); the yellow line is the mask that comes out with it. The ones that move pixels around move the mask with the image, and the ones that change colour, contrast or sharpness leave it where it was:</p>
<div class="cell" data-renderings="["light","dark"]">
<div class="cell-output-display light-content">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i1.wp.com/datascienceguts.com/posts/2026-11-12-segmentation-tour/index_files/figure-html/augmentations-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>The original image and fifteen albumentations transforms. Yellow: the outline of the mask after each transform.</figcaption>
</figure>
</div>
</div>
</div>
</section>
<section id="big-images-tiles-instead-of-shrinking" class="level2">
<h2 class="anchored" data-anchor-id="big-images-tiles-instead-of-shrinking">Big images: tiles instead of shrinking</h2>
<p>Every image is resized to <code>input_shape</code> before the network sees it. The nuclei lose little that way, which is why <code>nuclei.yaml</code> has no <code>splits</code>, but a 1024-pixel image shrunk to 128 keeps a nucleus as a few pixels. <strong><code>splits</code></strong> cuts the image into a grid of tiles instead, one number per spatial dimension:</p>
<pre>models:
  - name: tiled
    architecture: u_net
    input_shape: [128, 128]     # what the network sees: one tile
    splits: [4, 4]              # 4 rows x 4 columns
    batch_size: 2</pre>
<p>The constructors take the same field:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-8-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-8-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-8-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-8-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-8-1" class="tab-pane active" aria-labelledby="tabset-8-1-tab">
<div class="cell">
<pre>tiled &lt;- u_net(&quot;tiled&quot;, input_shape = c(128, 128), splits = c(4, 4),
               blocks = 4, filters = 16, batch_size = 2, epochs = 40)
platypus_spec(
  data = segmentation_data(&quot;nuclei&quot;, &quot;nuclei&quot;, colormap = binary_colormap),
  models = list(tiled)
)
#&gt; platypus specification
#&gt;   data        : nuclei 
#&gt;   classes     : 2 
#&gt;   models      : 1 
#&gt;     tiled          u_net            128x128, tiled 4x4, cce, 40 epochs</pre>
</div>
</div>
<div id="tabset-8-2" class="tab-pane" aria-labelledby="tabset-8-2-tab">
<pre>tiled = {&quot;name&quot;: &quot;tiled&quot;, &quot;architecture&quot;: &quot;u_net&quot;,
         &quot;input_shape&quot;: [128, 128], &quot;splits&quot;: [4, 4],
         &quot;blocks&quot;: 4, &quot;filters&quot;: 16, &quot;batch_size&quot;: 2, &quot;epochs&quot;: 40}
spec = pp.from_dict({
    &quot;task&quot;: &quot;semantic_segmentation&quot;,
    &quot;data&quot;: {&quot;train_path&quot;: &quot;nuclei&quot;, &quot;validation_path&quot;: &quot;nuclei&quot;,
             &quot;colormap&quot;: [[0, 0, 0], [255, 255, 255]]},
    &quot;models&quot;: [tiled],
})
m = spec.models[0]
m.load_shape, m.tiles_per_image
#&gt; ((512, 512), 16)</pre>
</div>
</div>
</div>
<p><code>input_shape</code> is still what the network sees, which is now one tile. The image is read at <code>splits</code> times that and cut, so one source image gives as many training samples as there are tiles:</p>
<table class="caption-top table">
<thead>
<tr class="header">
<th>splits</th>
<th style="text-align: right;">image read at</th>
<th style="text-align: right;">tiles per image</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>not set</td>
<td style="text-align: right;">128 x 128</td>
<td style="text-align: right;">1</td>
</tr>
<tr class="even">
<td><code>[2, 3]</code></td>
<td style="text-align: right;">256 x 384</td>
<td style="text-align: right;">6</td>
</tr>
<tr class="odd">
<td><code>[4, 4]</code></td>
<td style="text-align: right;">512 x 512</td>
<td style="text-align: right;">16</td>
</tr>
</tbody>
</table>
<div class="cell" data-renderings="["light","dark"]">
<div class="cell-output-display light-content">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i0.wp.com/datascienceguts.com/posts/2026-11-12-segmentation-tour/index_files/figure-html/tiling-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>One 1024 x 1024 image from the nuclei data as a network with input_shape 128 x 128 gets it: shrunk without splits, cut into sixteen tiles with splits [4, 4].</figcaption>
</figure>
</div>
</div>
</div>
<p>Three things follow:</p>
<ul>
<li>The source image does not have to be a multiple of the tile: it is resized to the size it is read at &#8211; 1024 x 1024 to 512 x 512 above, and a 696 x 520 image to the same square &#8211; and only then cut. The tile has the rule it had without tiling: every block halves it, so with <code>blocks = 4</code> each side of <code>input_shape</code> must divide by 16. A tile of <code>127 x 127</code> is refused when the file is read: <em>every spatial dimension must be divisible by 16; input_shape=(127, 127) is not</em>.</li>
<li><code>predict()</code> puts the tiles back together, so it returns one mask per image at its original size rather than sixteen pieces.</li>
<li><code>cldice</code> and <code>surface_dice</code> are properties of a whole mask, not of a tile, so a tiled run does not report them per epoch; <code>evaluate_cases()</code> scores them per image afterwards. A callback that watches one of them on a tiled run is refused when the file is read.</li>
</ul>
</section>
<section id="the-series-so-far" class="level2">
<h2 class="anchored" data-anchor-id="the-series-so-far">The series so far</h2>
<ol type="1">
<li><a href="https://datascienceguts.com/posts/2026-11-03-rplatypus/" rel="nofollow" target="_blank">rplatypus rebuilt: deep learning for medical images in R</a> &#8211; why the 2020 package had to be rewritten, how it now installs its own Python, and a first mask from published weights.</li>
<li><a href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Nuclei, five years later</a> &#8211; a mean Dice of 0.92, and the three images it hides.</li>
<li><strong>Semantic segmentation in rplatypus and pyplatypus: what is in the box</strong> &#8211; this post.</li>
</ol>
<p>The next one leaves segmentation for classification, where the number that hides things is not a mean but an accuracy: a classifier at 0.97 that has learnt nothing at all.</p>


</section>

 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/"> Data Science Guts</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/semantic-segmentation-in-rplatypus-and-pyplatypus-what-is-in-the-box/">Semantic segmentation in rplatypus and pyplatypus: what is in the box</a>]]></content:encoded>
					
		
		<enclosure url="https://datascienceguts.com/posts/2026-11-12-segmentation-tour/thumbnail.png" length="0" type="image/png" />

		<post-id xmlns="com-wordpress:feed-additions:1">404816</post-id>	</item>
		<item>
		<title>Bioconductor Carpentries workshops in Japanese: lessons from Japan</title>
		<link>https://www.r-bloggers.com/2026/11/bioconductor-carpentries-workshops-in-japanese-lessons-from-japan/</link>
		
		<dc:creator><![CDATA[Kozo Nishida]]></dc:creator>
		<pubDate>Thu, 12 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>An apology first<br />
This post should have been written a year ago. The two workshops it describes took place in August and September 2025, and a report in Japanese appeared in the newsletter of the Japanese Society for Bioinformatics (JSBi) in Jan...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/bioconductor-carpentries-workshops-in-japanese-lessons-from-japan/">Bioconductor Carpentries workshops in Japanese: lessons from Japan</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/"> Bioconductor community blog</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 





<section id="an-apology-first" class="level2">
<h2 class="anchored" data-anchor-id="an-apology-first">An apology first</h2>
<p>This post should have been written a year ago. The two workshops it describes took place in August and September 2025, and a report in Japanese appeared in the newsletter of the Japanese Society for Bioinformatics (JSBi) in January 2026. The report to the wider, English-speaking Bioconductor community is only arriving now, in November 2026.</p>
<p>I am sorry for the delay. I owe this apology to the participants, helpers and co-hosts who made the workshops happen, and to everyone in the Bioconductor community who has been curious about how teaching Bioconductor in a local language went in Japan. Part of the reason for the delay is explained later in this post, because it is tied to the future of our translation infrastructure.</p>
</section>
<section id="the-first-bioconductor-carpentries-workshops-taught-in-japanese" class="level2">
<h2 class="anchored" data-anchor-id="the-first-bioconductor-carpentries-workshops-taught-in-japanese">The first Bioconductor Carpentries workshops taught in Japanese</h2>
<p>Bioconductor has been a member organisation of <a href="https://carpentries.org/" rel="nofollow" target="_blank">The Carpentries</a> since August 2022 (<a href="https://blog.bioconductor.org/posts/2022-07-12-carpentries-membership/" rel="nofollow" target="_blank">announcement</a>). The Bioconductor Training Committee maintains a set of Carpentries-style <a href="https://training.bioconductor.org/#modules" rel="nofollow" target="_blank">lesson modules</a>, ranging from an introduction to R and Bioconductor to an advanced course on bulk RNA-seq analysis. How these lessons are developed and taught is described in <a href="https://doi.org/10.1371/journal.pcbi.1012925" rel="nofollow" target="_blank">Drnevich et al. (2025), <em>PLOS Computational Biology</em></a>.</p>
<p>Bringing this training to communities around the world in their own languages is supported by the Chan Zuckerberg Initiative (CZI) Essential Open Source Software for Science (EOSS) grant <a href="https://chanzuckerberg.com/eoss/proposals/delivering-high-quality-bioconductor-training-for-a-worldwide-community/" rel="nofollow" target="_blank"><em>Delivering High Quality Bioconductor Training for a Worldwide Community</em></a>.</p>
<p>As part of that effort, the Japanese Bioconductor community organised two two-day workshops in 2025, co-hosted with the <a href="https://www.jsbi.org/" rel="nofollow" target="_blank">Japanese Society for Bioinformatics (JSBi)</a>. To our knowledge these were the first Bioconductor workshops ever taught in Japanese. Both were hybrid events, with an in-person classroom and a Zoom connection for remote participants. I (Kozo Nishida) was the instructor for both. I serve on the Bioconductor Community Advisory Board and in the working group that translates Bioconductor teaching materials, and these workshops were the first time our Japanese translations were used in front of a class.</p>
<section id="workshop-1-introduction-to-r-and-bioconductor-osaka-56-august-2025" class="level3">
<h3 class="anchored" data-anchor-id="workshop-1-introduction-to-r-and-bioconductor-osaka-56-august-2025">Workshop 1: Introduction to R and Bioconductor (Osaka, 5–6 August 2025)</h3>
<p>The first workshop took place at the Taniguchi Memorial Hall in the Integrated Life Science Building of the Research Institute for Microbial Diseases, Osaka University, with Naohisa Goto as helper and local host.</p>
<ul>
<li><strong>Day 1</strong> followed the Japanese translation of <a href="https://bioconductor-translations.github.io/bioc-intro-ja/" rel="nofollow" target="_blank"><em>Introduction to data analysis with R and Bioconductor</em></a> (<code>bioc-intro</code>): how to organise data for analysis, then data manipulation and visualisation with the tidyverse packages, taught by live coding with every participant typing along.</li>
<li><strong>Day 2</strong> followed the Japanese translation of <a href="https://bioconductor-translations.github.io/bioc-project-ja/" rel="nofollow" target="_blank"><em>The Bioconductor project</em></a> (<code>bioc-project</code>): installing Bioconductor packages, getting help, S4 classes, biological sequences, <code>GenomicRanges</code>, annotation resources and <code>SummarizedExperiment</code>.</li>
</ul>
<p>The <a href="https://bioconductor-translations.github.io/2025-08-05-osaka/" rel="nofollow" target="_blank">workshop website</a> and <a href="https://bioconductor-translations.github.io/2025-08-05-osaka-slides/" rel="nofollow" target="_blank">slides</a> are online, in Japanese.</p>
</section>
<section id="workshop-2-rna-seq-analysis-with-bioconductor-kyoto-1112-september-2025" class="level3">
<h3 class="anchored" data-anchor-id="workshop-2-rna-seq-analysis-with-bioconductor-kyoto-1112-september-2025">Workshop 2: RNA-seq analysis with Bioconductor (Kyoto, 11–12 September 2025)</h3>
<p>The second workshop was the advanced course, held at the Inamori Memorial Hall of Kyoto Prefectural University, with Atsushi Fukushima as helper and local host. We used the Japanese translation of <a href="https://bioconductor-translations.github.io/bioc-rnaseq-ja/" rel="nofollow" target="_blank"><em>RNA-seq analysis with Bioconductor</em></a> (<code>bioc-rnaseq</code>) and live-coded the whole bulk RNA-seq workflow, apart from the read quantification step that produces the count matrix.</p>
<ul>
<li><strong>Day 1</strong> covered an introduction to RNA-seq, RStudio projects and the experimental data, importing quantified data into R and annotating it, and exploratory analysis and quality control, everything that comes before differential expression analysis.</li>
<li><strong>Day 2</strong> covered differential expression analysis, a closer look at design matrices, and gene set enrichment analysis for interpreting the results.</li>
</ul>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="https://i0.wp.com/blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/kyoto-workshop.jpg?ssl=1" class="lightbox" data-gallery="quarto-lightbox-gallery-1" title="The second workshop, at Kyoto Prefectural University." rel="nofollow" target="_blank"><img src="https://i0.wp.com/blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/kyoto-workshop.jpg?w=578&#038;ssl=1" class="img-fluid quarto-figure quarto-figure-center figure-img" alt="A lecture room at Kyoto Prefectural University during the second workshop. Participants sit at long desks with laptops while the instructor stands at the front next to a screen showing R code." data-recalc-dims="1"></a></p>
</figure>
</div>
<figcaption>The second workshop, at Kyoto Prefectural University.</figcaption>
</figure>
</div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><a href="https://i2.wp.com/blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/bioc-rnaseq-ja.jpg?ssl=1" class="lightbox" data-gallery="quarto-lightbox-gallery-2" title="The Japanese translation of the RNA-seq analysis with Bioconductor lesson, used in the second workshop." rel="nofollow" target="_blank"><img src="https://i2.wp.com/blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/bioc-rnaseq-ja.jpg?w=578&#038;ssl=1" class="img-fluid quarto-figure quarto-figure-center figure-img" alt="Screenshot of the Japanese translation of the RNA-seq analysis with Bioconductor lesson, showing the gene set enrichment analysis episode with its questions and objectives in Japanese." data-recalc-dims="1"></a></p>
</figure>
</div>
<figcaption>The Japanese translation of the <em>RNA-seq analysis with Bioconductor</em> lesson, used in the second workshop.</figcaption>
</figure>
</div>
<p>The <a href="https://bioconductor-translations.github.io/2025-09-11-kyoto/" rel="nofollow" target="_blank">workshop website</a> and <a href="https://bioconductor-translations.github.io/2025-09-11-kyoto-slides/" rel="nofollow" target="_blank">slides</a> are online as well.</p>
</section>
<section id="who-took-part" class="level3">
<h3 class="anchored" data-anchor-id="who-took-part">Who took part</h3>
<table class="caption-top table">
<thead>
<tr class="header">
<th>Workshop</th>
<th>In person</th>
<th>Online</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Osaka, August 2025</td>
<td>about 3</td>
<td>about 45</td>
</tr>
<tr class="even">
<td>Kyoto, September 2025</td>
<td>about 6</td>
<td>about 45</td>
</tr>
</tbody>
</table>
<p>Attendance fluctuated over the two long days, so these are approximate numbers. Most participants joined online, which reflects how geographically spread out the Japanese bioinformatics community is. The hybrid format clearly lowered the barrier to attending.</p>
<p>We are grateful to JSBi for co-hosting, to the venue hosts at Osaka University and Kyoto Prefectural University, and above all to everyone who took part. We intend to keep running and improving these workshops. Our hope is that participants will not only learn from them but go on to teach, so that the circle keeps widening.</p>
<p>A report on both workshops, written in Japanese together with Naohisa Goto and Atsushi Fukushima, appeared in <a href="https://www.jsbi.org/media/files/_u/topic/file/nl48.pdf" rel="nofollow" target="_blank">JSBi Newsletter No. 48 (January 2026, PDF in Japanese)</a>.</p>
</section>
</section>
<section id="where-the-materials-live" class="level2">
<h2 class="anchored" data-anchor-id="where-the-materials-live">Where the materials live</h2>
<p>Everything we used is in the <a href="https://github.com/bioconductor-translations" rel="nofollow" target="_blank">bioconductor-translations</a> GitHub organisation, and anyone is welcome to use it for their own teaching:</p>
<ul>
<li>Japanese lesson translations: <a href="https://github.com/bioconductor-translations/bioc-intro-ja" rel="nofollow" target="_blank">bioc-intro-ja</a>, <a href="https://github.com/bioconductor-translations/bioc-project-ja" rel="nofollow" target="_blank">bioc-project-ja</a> and <a href="https://github.com/bioconductor-translations/bioc-rnaseq-ja" rel="nofollow" target="_blank">bioc-rnaseq-ja</a>.</li>
<li>Workshop websites and slides: <a href="https://github.com/bioconductor-translations/2025-08-05-osaka" rel="nofollow" target="_blank">2025-08-05-osaka</a>, <a href="https://github.com/bioconductor-translations/2025-08-05-osaka-slides" rel="nofollow" target="_blank">2025-08-05-osaka-slides</a>, <a href="https://github.com/bioconductor-translations/2025-09-11-kyoto" rel="nofollow" target="_blank">2025-09-11-kyoto</a> and <a href="https://github.com/bioconductor-translations/2025-09-11-kyoto-slides" rel="nofollow" target="_blank">2025-09-11-kyoto-slides</a>.</li>
<li>The organisation also hosts the French translation <a href="https://bioconductor-translations.github.io/bioc-intro-fr/" rel="nofollow" target="_blank">bioc-intro-fr</a>, which was used in the <a href="https://blog.bioconductor.org/posts/2025-12-11-benin-course/" rel="nofollow" target="_blank">Bioconductor course in Benin</a>.</li>
</ul>
</section>
<section id="why-this-post-is-late-and-what-i-have-learned-about-our-translation-setup" class="level2">
<h2 class="anchored" data-anchor-id="why-this-post-is-late-and-what-i-have-learned-about-our-translation-setup">Why this post is late, and what I have learned about our translation setup</h2>
<p>Here is the honest part. One reason this post took so long is that I have not been able to maintain <a href="https://bioconductor.crowdin.com/" rel="nofollow" target="_blank">bioconductor.crowdin.com</a> and the <a href="https://github.com/bioconductor-translations" rel="nofollow" target="_blank">bioconductor-translations</a> repositories the way they deserve, and I felt guilty writing a cheerful report about workshops while the infrastructure behind them was falling behind.</p>
<p>Some background. I was the one who proposed that the Bioconductor community use Crowdin to manage translations. The model was <a href="https://turingway.crowdin.com/" rel="nofollow" target="_blank">The Turing Way</a>, which translates its handbook on Crowdin with great success. Crowdin’s support for open-source projects has been generous, and I remain very grateful for it.</p>
<p>However, the two projects differ in an important way. The Turing Way’s source documents are plain Markdown. The Carpentries lessons are written in R Markdown, with executable code chunks and lesson-specific fenced blocks for callouts, challenges and solutions. To feed them into Crowdin we had to convert R Markdown to Markdown into separate translation repositories, sync the translated strings back, and reassemble buildable lessons from the result. The Crowdin and GitHub integration that this requires is complex and fragile, and it is not easy to manage.</p>
<p>At the same time, the world has changed since I made that proposal. AI translation quality has improved substantially, and with <a href="https://agentskills.io/" rel="nofollow" target="_blank">Agent Skills</a> an AI agent can now produce a first translation according to project rules: a shared glossary, consistent terminology, and clear instructions about what to leave untouched, such as code, package names and function names. The resulting translation can then be reviewed by community members with appropriate language and domain expertise. Under these conditions, managing translations directly in GitHub, with translated lessons living alongside their sources and reviewed through pull requests, is more efficient than keeping a Crowdin and GitHub integration alive.</p>
<p>So, in order to make Bioconductor teaching materials available in more languages, I no longer think we should keep using <a href="https://bioconductor.crowdin.com/" rel="nofollow" target="_blank">bioconductor.crowdin.com</a>, and I intend to retire it. This is not a complaint about Crowdin. It is a recognition that the tool was the right one for a different kind of source document, and that a simpler workflow now serves our lessons better.</p>
</section>
<section id="what-comes-next" class="level2">
<h2 class="anchored" data-anchor-id="what-comes-next">What comes next</h2>
<p>I am setting up new, GitHub-only translation repositories together with an Agent Skill for rule-following AI translation, its documentation, and the outreach needed to bring more languages and more contributors on board. This work is supported by the <a href="https://bioconductor.org/developers/developer-champions-program/" rel="nofollow" target="_blank">Bioconductor Developer Champions Program</a>.</p>
<p>Please give me a little more time before the formal announcement. When it is ready, I will share it here on the blog.</p>
</section>
<section id="get-involved" class="level2">
<h2 class="anchored" data-anchor-id="get-involved">Get involved</h2>
<ul>
<li>Teach with the materials. The Japanese lessons, the workshop websites and the slides linked above are openly licensed and free for anyone to use.</li>
<li>If you are interested in translating Bioconductor teaching materials into your own language, watch the <a href="https://github.com/bioconductor-translations" rel="nofollow" target="_blank">bioconductor-translations</a> organisation for the upcoming announcement.</li>
<li>Join the Bioconductor community through the <a href="https://www.bioconductor.org/help/support/" rel="nofollow" target="_blank">support site</a> and the <a href="https://chat.bioconductor.org/" rel="nofollow" target="_blank">community chat</a>.</li>
</ul>
</section>
<section id="acknowledgements" class="level2">
<h2 class="anchored" data-anchor-id="acknowledgements">Acknowledgements</h2>
<p>Thank you to Naohisa Goto and Atsushi Fukushima for helping at and hosting the workshops, to JSBi for co-hosting them, and to every participant. This work was supported by the <a href="https://chanzuckerberg.com/eoss/proposals/delivering-high-quality-bioconductor-training-for-a-worldwide-community/" rel="nofollow" target="_blank">Chan Zuckerberg Initiative EOSS program</a> and by the <a href="https://bioconductor.org/developers/developer-champions-program/" rel="nofollow" target="_blank">Bioconductor Developer Champions Program</a>.</p>


</section>

<p>
© 2026 Bioconductor. Content is published under <a href="https://creativecommons.org/licenses/by/4.0/" rel="nofollow" target="_blank">Creative Commons CC-BY-4.0 License</a> for the text and <a href="https://opensource.org/licenses/BSD-3-Clause" rel="nofollow" target="_blank">BSD 3-Clause License</a> for any code. | <a href="https://www.r-bloggers.com/" rel="nofollow" target="_blank">R-Bloggers</a>
</p> 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/"> Bioconductor community blog</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/bioconductor-carpentries-workshops-in-japanese-lessons-from-japan/">Bioconductor Carpentries workshops in Japanese: lessons from Japan</a>]]></content:encoded>
					
		
		<enclosure url="https://blog.bioconductor.org/posts/2026-11-12-japanese-carpentries-workshops/kyoto-workshop.jpg" length="0" type="image/jpeg" />

		<post-id xmlns="com-wordpress:feed-additions:1">404813</post-id>	</item>
		<item>
		<title>Preparing the Data: Cattell&#8217;s 16 Factor Personality Measure</title>
		<link>https://www.r-bloggers.com/2026/11/preparing-the-data-cattells-16-factor-personality-measure/</link>
		
		<dc:creator><![CDATA[Unknown]]></dc:creator>
		<pubDate>Wed, 11 Nov 2026 19:25:15 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">http://www.r-bloggers.com/?guid=858f6d97921564f156dd5f9fc5b3b9de</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; "> As I said in my previous post, I'm working on a 2-part series where I show how to conduct psychometric analysis on a personality measure. Those posts are coming very soon, but today, I wanted to start by demonstrating how I prepare the data for psychom...</div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/preparing-the-data-cattells-16-factor-personality-measure/">Preparing the Data: Cattell’s 16 Factor Personality Measure</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="http://www.deeplytrivial.com/2026/11/preparing-data-cattells-16-factor.html"> Deeply Trivial</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
As I said in my <a href="https://www.deeplytrivial.com/2026/08/im-baaaaack.html" rel="nofollow" target="_blank">previous post</a>, I&#8217;m working on a 2-part series where I show how to conduct psychometric analysis on a personality measure. Those posts are coming very soon, but today, I wanted to start by demonstrating how I prepare the data for psychometric analysis.<div><br /></div>
So let&#8217;s start working with the Cattell&#8217;s 16 Factor Personality Measure, or more specifically, items from the International Personality Item Pool (IPIP) selected to reflect the 16 factors identified by Cattell. I happened upon this measure while exploring the <a href="https://openpsychometrics.org/" rel="nofollow" target="_blank">Open-Source Psychometrics Project website</a>, which not only allows you to take a wide variety of personality measures, it also makes the data collected through the website <a href="https://openpsychometrics.org/_rawdata/" rel="nofollow" target="_blank">freely available.</a> This would be a great source to use for a class project, someone learning psychometrics or working with larger datasets&#8230; or a blog post!<div><br /></div>
When you download the data, you&#8217;ll also get the measure&#8217;s codebook included in the zip file. However, the codebook is really a data dictionary of the items in the dataset (name, definition, and how it&#8217;s coded), and doesn&#8217;t tell you which items are reverse-scored. So I also recommended this <a href="https://ipip.ori.org/new16PFKey.htm" rel="nofollow" target="_blank">page</a>, which lists items of each measure in order, showing which are reverse-keyed, as well as the alpha for each scale from the initial research. You may also want to check out <a href="https://ipip.ori.org/new16PFTable.htm" rel="nofollow" target="_blank">this page</a> for additional details about the measure.<div><br /></div>
When conducting psychometric analysis, you may or may not need to do any data cleaning. It really depends on how the data were collected &#8211; was it in a controlled setting, like a testing center? Or less controlled like &#8211; in the case of this measure &#8211; an online survey? We likely have missing data, since none of the items were required. We could also have straight-line answering (where a person selects the same response for each item), or nonsensical answers to the open-response age question, such as one respondent who entered 2,147,483,647, which would put their birth right in the middle of the Paleoproterozoic era, when the only living beings on the planet were ones who didn&#8217;t require oxygen. <div><br /></div>
So let&#8217;s load our data and get started with cleaning.
<pre>library(tidyverse)
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## &#x2714; dplyr     1.2.1     &#x2714; readr     2.2.0
## &#x2714; forcats   1.0.1     &#x2714; stringr   1.6.0
## &#x2714; ggplot2   4.0.3     &#x2714; tibble    3.3.1
## &#x2714; lubridate 1.9.5     &#x2714; tidyr     1.3.2
## &#x2714; purrr     1.2.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## &#x2716; dplyr::filter() masks stats::filter()
## &#x2716; dplyr::lag()    masks stats::lag()
## &#x2139; Use the conflicted package (&lt;http://conflicted.r-lib.org/&gt;) to force all conflicts to become errors
library(ggthemes)

PF16 &lt;- read_tsv(&quot;data.csv&quot;)
## Rows: 49159 Columns: 169
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: &quot;\t&quot;
## chr   (1): country
## dbl (168): A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, B1, B2, B3, B4, B5, B6, ...
## 
## &#x2139; Use `spec()` to retrieve the full column specification for this data.
## &#x2139; Specify the column types or set `show_col_types = FALSE` to quiet this message.</pre>
According to documentation from the website, they removed anyone under the age of 13 from the downloadable set, as well as anyone who didn't want their data included. But there are still children in the set (13 to 17), and also our immortal beings (i.e., 2 billion years old). We definitely want to remove those rows from data, since a random answer to the age question could also mean random answers on the rest of the measure. So let's get some descriptive data on our age measure, and make decisions about who to keep and who to filter out.<div><br /></div>
<pre>PF16 %&gt;%
  filter(age &gt; 99) %&gt;%
  group_by(age) %&gt;%
  summarise(count = n())
## # A tibble: 63 × 2
##      age count
##    &lt;dbl&gt; &lt;int&gt;
##  1   100     5
##  2   101     2
##  3   102     1
##  4   118     1
##  5   141     1
##  6   147     1
##  7   158     1
##  8   181     1
##  9   188     1
## 10   193     1
## # &#x2139; 53 more rows</pre>
I did some Googling, and the oldest person to ever live (recorded) was 122 years, and the oldest person still living is 117. I’m inclined to say anyone over 102 is not being truthful, so we’ll plan on filtering them out. Likely many people were not being truthful about age, but unless we have direct evidence they're lying, we'll have to leave them in. With the remaining respondents, we’ll classify their ages into groups: 13 to 17, 18 to 24, 25 to 34, 35 to 44, 45 to 64, 65+. I’m also going to add IDs to the file. Since each row is a person, I’ll just use row numbers.
<pre>PF16 &lt;- PF16 %&gt;%
  filter(age &lt;= 102) %&gt;%
  mutate(ID = row_number(),
         age_group = cut(age,
                         breaks = c(12, 17, 24, 34, 44, 64, Inf),
                         labels = c(&quot;13 to 17&quot;, &quot;18 to 24&quot;, &quot;25 to 34&quot;, &quot;35 to 44&quot;, &quot;45 to 64&quot;, &quot;65+&quot;),
                         right = TRUE))</pre>
To create the age grouping variable, I used the cut function, where I define my break points and use right = TRUE to say that the value provided is the end of each break. The first value in the "right = TRUE" notation should be the starting value minus 1.<div><br /></div>
Let's also replace missing values (currently coded with 0) with NA. Since all of our items are in a row, we can use mutate(across(firstvariable:lastvariable)) in the string we want to recode.
<pre>PF16 &lt;- PF16 %&gt;%
  mutate(across(A1:P10, ~na_if(., 0)))</pre>
Now we want to start identifying people with response patterns that won't provide meaningful information. This would include people who responded carelessly - like selecting the same option for each item or going between two options (like 4-5-4-5) - or with too much missing data. However, we want to be conservative in who we remove. Presumably, this was a volunteer survey that people did out of interest or curiosity. In those cases, some careless responding may mean the item was confusing, doesn't measure what it's supposed to, etc. But that doesn't mean there might not be some captive audience respondents - people who were required to complete the measure as part of coursework or a group activity - that just wanted to get it over with. Without being able to measure or control the context, we just don't know. This is why I'd like to be conservative in who I remove, because I'd rather retain people who were truly being careless than risk removing people whose "careless" responses are actually due to problematic items or scales.<div><br /></div>
There are a few ways we can measure careless responses. One is by looking at their longest string of identical responses. The other way is with intra-individual response variability, which is better at detecting people who stick to two consecutive options. The r package, careless, can do all of this for us. Reminder: if you don't have a particular package, you'll need to install it first with install.packages("packagename") before loading it. The longstring function will provide the count of the longest string of repeated answers; adding avg = TRUE will also give us the average length of strings across the entire row. irv computes intra-individual response variability, or the amount of variation in each row's responses. All of this will be easier if we have a dataset containing only item responses, and we generate a new dataset with the data from the careless functions; that way, we won't inadvertently use demographics, IDs, or results from a previous function. I'll also add to my flag dataset the number of missing responses and the longest string divided by total number of items, to measure the percentage of their responses included in that string.
<pre>library(careless)

Items &lt;- PF16 %&gt;%
  select(A1:P10)

longstrings &lt;- longstring(Items, avg = TRUE)

Flags &lt;- tibble(
  ID = PF16$ID,
  longstring = longstrings$longstr,
  avgstring = longstrings$avgstr,
  irv = irv(Items, na.rm = TRUE),
  na_count = rowSums(is.na(Items)),
  pct_long = longstring/163
)</pre>
Let's start with IRV, which should help us identify the worst offenders in terms of careless responding. Lower IRVs (say, less than 0.50) would identify people who gave the same answer or one of two answers to every item. Let's visualize our irv variable before we make any decisions on what to filter out.
<pre>library(ggthemes)
Flags %&gt;% 
  ggplot(aes(irv)) + 
  geom_histogram() +
  theme_economist_white()
## `stat_bin()` using `bins = 30`. Pick better value `binwidth`.
## Warning: Removed 171 rows containing non-finite outside the scale range
## (`stat_bin()`).</pre>
<div class="separator" style="clear: both;"><a href="https://i1.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggjv8Ju-QG2dUlI4U5UFMy6_2CiCKuMTshmBPnEZzvfoNmO6d8clt_TdG2O2fjLnitdlJsSAWU0vaUx_h70HscJQ4nLPSCY_seiA0NTEkq2NVSKx84verdwiLOv15EHSJGd3iPLdXurVWxlPtZgawfYvj6p61m2lC_dOnnYQuKATqhyphenhyphenDgrFxU_TSVzlVAZ/s1558/irv%20histogram.png?ssl=1" style="display: block; padding: 1em 0px; text-align: center;" rel="nofollow" target="_blank"><img loading="lazy" alt="" border="0" data-original-height="1064" data-original-width="450" src="https://i2.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggjv8Ju-QG2dUlI4U5UFMy6_2CiCKuMTshmBPnEZzvfoNmO6d8clt_TdG2O2fjLnitdlJsSAWU0vaUx_h70HscJQ4nLPSCY_seiA0NTEkq2NVSKx84verdwiLOv15EHSJGd3iPLdXurVWxlPtZgawfYvj6p61m2lC_dOnnYQuKATqhyphenhyphenDgrFxU_TSVzlVAZ/w400-h274/irv%20histogram.png?resize=450%2C1064&#038;ssl=1" width="450" data-recalc-dims="1" /></a></div>
We have a number of cases with an IRV of 0. Those are our true straight-liners. There are also cases with IRV of 0.50 or less. Those would be the people who bounce between two options. The overall counts are low, so we won't lose too many cases by filtering out any with an IRV of 0.50 or less.
<pre>Flags &lt;- Flags %&gt;%
  filter(irv &gt; 0.50)</pre>
This dropped 381 cases. Now let's look at longstrings. Our IRV filter dropped people who showed careless response patterns across the whole measure, but we likely still have careless responders here. We just need to be judicious and data-driven in how we identify them.
<pre>range(Flags$pct_long)
## [1] 0.006134969 0.441717791</pre>
I also ran a basic histogram much like the one above for IRV (not pictured), and counts above 10% were so low, they weren't even visible. I reran the histogram, this time specifying bin width to figure out an exact cutoff, and limiting the scale to values between 0 and 50% (since we know our longest string is 44%).
<pre>Flags %&gt;%
  ggplot(aes(pct_long)) +
  geom_histogram(binwidth = .01) +
  scale_x_continuous(limits = c(0, 0.50)) +
  theme_economist_white()
  </pre>
<div class="separator" style="clear: both;"><a href="https://i0.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_Z7gPdZPP3Od4Vy6WPC6uDSE46vHz1yCYfJvdvDrUCDOUgf-GKtp8804S79JVQ2b71oD68IdJtVQeEhTOormr5FXJaR9b9mjlqylQlqKVInLq176pVtOJWtDuPwaiR0m1OmP8EsqHhd7wvp49YQdj_XXgU9y59rOowI2PppQZXNQCOB2IUNzp6bLdUhXb/s1558/pct_long%20histogram.png?ssl=1" style="display: block; padding: 1em 0px; text-align: center;" rel="nofollow" target="_blank"><img alt="" border="0" data-original-height="1064" data-original-width="450" src="https://i1.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_Z7gPdZPP3Od4Vy6WPC6uDSE46vHz1yCYfJvdvDrUCDOUgf-GKtp8804S79JVQ2b71oD68IdJtVQeEhTOormr5FXJaR9b9mjlqylQlqKVInLq176pVtOJWtDuPwaiR0m1OmP8EsqHhd7wvp49YQdj_XXgU9y59rOowI2PppQZXNQCOB2IUNzp6bLdUhXb/s400/pct_long%20histogram.png?resize=450%2C1064&#038;ssl=1" width="450" data-recalc-dims="1" /></a></div>
<p>We have a large dropoff around the 9% string percentage. Let’s see
how many cases we’d lose with that cutoff, as well as 10%.</p>
<pre>Flags %&gt;%
  filter(pct_long &gt;= 0.09)
## # A tibble: 335 × 6
##       ID longstring avgstring   irv na_count pct_long
##    &lt;int&gt;      &lt;dbl&gt;     &lt;dbl&gt; &lt;dbl&gt;    &lt;dbl&gt;    &lt;dbl&gt;
##  1    42         15      1.90 0.553        0   0.0920
##  2    56         17      1.75 0.860        0   0.104 
##  3   176         15      1.96 0.769        1   0.0920
##  4   553         17      2.51 0.650        0   0.104 
##  5   617         17      2.51 0.737       11   0.104 
##  6  2038         17      3.40 0.688        0   0.104 
##  7  3418         18      2.09 1.31         0   0.110 
##  8  3995         15      1.47 1.30         1   0.0920
##  9  4035         15      2.86 0.985        0   0.0920
## 10  4385         16      2.12 0.849        0   0.0982
## # &#x2139; 325 more rows
Flags %&gt;%
  filter(pct_long &gt;= 0.10)
## # A tibble: 265 × 6
##       ID longstring avgstring   irv na_count pct_long
##    &lt;int&gt;      &lt;dbl&gt;     &lt;dbl&gt; &lt;dbl&gt;    &lt;dbl&gt;    &lt;dbl&gt;
##  1    56         17      1.75 0.860        0    0.104
##  2   553         17      2.51 0.650        0    0.104
##  3   617         17      2.51 0.737       11    0.104
##  4  2038         17      3.40 0.688        0    0.104
##  5  3418         18      2.09 1.31         0    0.110
##  6  6013         19      1.68 0.805        0    0.117
##  7  6909         21      1.92 1.51         1    0.129
##  8  7120         17      2.76 0.574        2    0.104
##  9  7158         18      1.99 0.718        6    0.110
## 10  7686         18      2.04 0.706        0    0.110
## # &#x2139; 255 more rows</pre>
A cutoff of 9% translate to a longest string of 15, and 10% translates to 17. Most of our scales contain 10 items, with the exception being scale B, intellect, at 13 items. Further, each scale contains 3-5 negatively worded items. If those items and scales are functioning as expected, the longest possible string would be 8. But what if those items and scales <b>aren't</b> functioning as expected? What if a negative item, that test developers thought represented the opposite of the trait being measured, is completely unrelated?<div><br /></div><div>At the same time, these groups make up a small proportion of the overall sample. Using a cutoff of 10%, we would only lose about 0.5% of our overall total. So let's use 10%.</div>
<pre>Flags &lt;- Flags %&gt;%
  filter(pct_long &lt; 0.1)
NAs &lt;- Flags %&gt;% 
  group_by(na_count) %&gt;% 
  summarise(count = n())

head(NAs)
## # A tibble: 6 × 2
##   na_count count
##      &lt;dbl&gt; &lt;int&gt;
## 1        0 35127
## 2        1  8705
## 3        2  2595
## 4        3   903
## 5        4   423
## 6        5   158</pre>
If we kept only complete cases, we’d still have over 35,000 cases to
work with. But our psychometric approaches (CTT and Rasch) can handle
some (but not a lot of) missing data. Using a cutoff of 2 missing or fewer would let us keep most of our sample, and shouldn't impact our ability to validate these scales.
<pre>Flags &lt;- Flags %&gt;%
  filter(na_count &lt;= 2)</pre>
Now that we have our sample of people who appear to have put some
effort into responding to the items, we can filter our PF16 dataset to
just include those individuals.
<pre>PF16 &lt;- PF16 %&gt;%
  filter(ID %in% Flags$ID)</pre>
And to make it easier on myself, I'll save this dataset, so that I can start with it in my next post, where we'll work on CTT psychometric analysis!
<pre>write_csv(PF16, &quot;Cleaned PF16 Data.csv&quot;)</pre>
See you in my next post!

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.deeplytrivial.com/2026/11/preparing-data-cattells-16-factor.html"> Deeply Trivial</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/preparing-the-data-cattells-16-factor-personality-measure/">Preparing the Data: Cattell’s 16 Factor Personality Measure</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404804</post-id>	</item>
		<item>
		<title>Distribution of race by ZCTA</title>
		<link>https://www.r-bloggers.com/2026/11/distribution-of-race-by-zcta/</link>
		
		<dc:creator><![CDATA[Jerry Tuttle]]></dc:creator>
		<pubDate>Wed, 11 Nov 2026 19:16:34 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">http://www.r-bloggers.com/?guid=4d070045b9ca653b7e5c5359e7cedd8d</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>What's a ZCTA?</p>
<p>     <br />
    For years I assumed that a city consists of a number of zipcodes, and the city's population was simply the sum of the populations those zipcodes. </p>
<p>     <br />
    It turns ...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/distribution-of-race-by-zcta/">Distribution of race by ZCTA</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://onlinecollegemathteacher.blogspot.com/2026/11/distribution-of-race-by-zcta.html"> Online College Math Teacher</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<font size = 3>
  
<h3>What&#8217;s a ZCTA?</h3>
  
      
    For years I assumed that a city consists of a number of zipcodes, and the city&#8217;s population was simply the sum of the populations those zipcodes. <p>
  
     
    It turns out this isn&#8217;t quite right. <p>
  
       
    Zipcodes are postal delivery routes. They are a collection of lines (streets) and points (delivery boxes) designed purely for mail trucks to drop off letters as efficiently as possible. They do not have official, enclosed borders.  Also, zipcode definitions frequently change. <p>
  
      
    To make geographical analysis possible, the Census Bureau invented ZCTAs (ZIP Code Tabulation Areas) using Census blocks into artificial, enclosed geographic polygons. Each resulting shape is given a 5-digit name that matches the dominant zipcode of the greatest number of addresses inside it.  <p>
  
     
    The U.S. Census Bureau publishes decennial population by city, and also by these smaller ZCTAs, but never by actual USPS zipcodes. (They do publish other statistics by zipcode.)<p>
  
     
    A ZCTA polygon is unlikely to perfectly match the geographical area of the dominant zipcode. Moreover, a ZCTA may actually cross a municipal boundary.  For example, ZCTA 11001 (Floral Park) and 11040 (New Hyde Park) span both Queens (NYC) and Nassau Counties, but most of their area and population are in Nassau County. So counting population in such ZCTAs in full for NYC would overstate NYC&#8217;s population. Adding the populations of all the ZCTAs is not going to equal the population of NYC.<p>
  
<h3>Distribution of race in a smaller city</h3>
  
     
  
   I am using the R package tidycensus to pull data from the U.S. Census Bureau API into R.
Census data includes many possible variables.  Some broad categories are population, median household income, housing units, median rent, education, occupation, etc.  For convenience I chose population by race for this project, although I could have chosen something else.  You need to request a one-time Census API key from <a href="https://api.census.gov/data/key_signup.html" rel="nofollow" target="_blank">
  https://api.census.gov/data/key_signup.html</a> .<p>  
  
     
   With the above as background, I decided to examine the distribution of race in a smaller city, Fort Myers, Florida.  There are 13 inhabited ZCTAs in the greater Fort Myers area, plus some miscellaneous ZCTAs such as for post office boxes.  A map of the greater Fort Myers area with its ZCTAs, which extend beyond Fort Myers City, is as follows, where the dashed red line indicates Fort Myers City limits: <p>
  
<div class="separator" style="clear: both;"><a href="https://i0.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwj05xDaBKKfwyqJiuKxHwbetVwgHwahcwPCwz5kvm_nitLHD9XQm820Br-RZ9CpLWgc7PGOFU8gIqeG3jrkV82jhhDgRXkT8Fu0HlxgdtalX6yFOLwbbjlO7uTXex3tUdhpkSvvvhnmQIZdOaW6hhmMjY9RvFcWhca2bY5TUnk88FdAwEZdhOTWg1YLs/s579/map1.png?ssl=1" style="display: block; padding: 1em 0; text-align: center; " rel="nofollow" target="_blank"><img alt="" border="0" width="400" data-original-height="438" data-original-width="450" src="https://i2.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwj05xDaBKKfwyqJiuKxHwbetVwgHwahcwPCwz5kvm_nitLHD9XQm820Br-RZ9CpLWgc7PGOFU8gIqeG3jrkV82jhhDgRXkT8Fu0HlxgdtalX6yFOLwbbjlO7uTXex3tUdhpkSvvvhnmQIZdOaW6hhmMjY9RvFcWhca2bY5TUnk88FdAwEZdhOTWg1YLs/s400/map1.png?resize=400%2C438&#038;ssl=1" data-recalc-dims="1"/></a></div>
  
  
      
  I used the classification of 2020 Census Table P2 data, which separates Hispanic people of any race from  non-Hispanic people who are then classified by race. These are mutually exclusive categories.  (I recognize some awkwardness to this, and I am not attempting to describe every racial and ethnic category.  For convenience I will use the word &#8220;race&#8221;, and I apologize if this offends anyone; Hispanic is not a race.)  I combined five smaller non-Hispanic races including Multiracial into an Other category.  The following is the distribution of race of Fort Myers City, which had a 2020 population of 86,395:<p>

<center>
<table>
  <table border=1>
  <tr>
    <th>Race (NH = Non-Hispanic)</th>
    <th>Percentage</th>
  </tr>
  <tr>
    <td>White (NH)</td>
    <td>47.51%</td>
  </tr>
  <tr>
    <td>Hispanic (any race)</td>
    <td>23.61%</td>
  </tr>
  <tr>
    <td>Black (NH)</td>
    <td>21.87%</td>
  </tr>
  <tr>
    <td>Other (NH)</td>
    <td>7.01%</td>
  </tr>  
</table>
</center><p>
  
     
  Superimposing the above table as a pie chart onto the map of Fort Myers City gives the following:                                                                                               
<div class="separator" style="clear: both;"><a href="https://i1.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm-L5W-2mZ6APPjqXIWUrfvjMQbROyhOMxHRCH-B0sXdyxHAoNMDAuSOF1CcN4X8QHbAA6lyZqsGF3pWHLg6P-yl_WCq5-w4tpiGkZgjRfGGcl6-eDVj9ktDHhc0Sr_BiQm5vNQtWVIwqRt4zVB42H9JmPqG8CCdgLmbqrgFUYUqV6DB-XWlHaGzmgQX0/s579/map2.png?ssl=1" style="display: block; padding: 1em 0; text-align: center; " rel="nofollow" target="_blank"><img alt="" border="0" width="400" data-original-height="438" data-original-width="450" src="https://i0.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgm-L5W-2mZ6APPjqXIWUrfvjMQbROyhOMxHRCH-B0sXdyxHAoNMDAuSOF1CcN4X8QHbAA6lyZqsGF3pWHLg6P-yl_WCq5-w4tpiGkZgjRfGGcl6-eDVj9ktDHhc0Sr_BiQm5vNQtWVIwqRt4zVB42H9JmPqG8CCdgLmbqrgFUYUqV6DB-XWlHaGzmgQX0/s400/map2.png?resize=400%2C438&#038;ssl=1" data-recalc-dims="1"/></a></div>
  
     
   Next, I collected the data and drew the pie chart for a single ZCTA, 33905.  This ZCTA has a greater percentage of Hispanics than Fort Myers City, and a smaller percentage of blacks.
  
<div class="separator" style="clear: both;"><a href="https://i1.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOnQ77MtvCPgWwbNElEpHw4YKi2r8_nUINQVSCO0ORmDrTYeCbUlx0vsFhqDT0UmJckl68-zHXv9tXFaWkw4COykSvcrnvBtw_j-OZrhP0xX5ntt5fsu9ENupyE4nIBznmK5rNLEjN7_3lX0ahGfXXfQGzYGRick54iOwqyjbylKk5iXe3EFI1IMEgwwc/s579/map3.png?ssl=1" style="display: block; padding: 1em 0; text-align: center; " rel="nofollow" target="_blank"><img alt="" border="0" width="400" data-original-height="438" data-original-width="450" src="https://i1.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOnQ77MtvCPgWwbNElEpHw4YKi2r8_nUINQVSCO0ORmDrTYeCbUlx0vsFhqDT0UmJckl68-zHXv9tXFaWkw4COykSvcrnvBtw_j-OZrhP0xX5ntt5fsu9ENupyE4nIBznmK5rNLEjN7_3lX0ahGfXXfQGzYGRick54iOwqyjbylKk5iXe3EFI1IMEgwwc/s400/map3.png?resize=400%2C438&#038;ssl=1" data-recalc-dims="1"/></a></div>
  
     
    I initially tried to create a single map with a separate pie chart for every ZCTA, but that map was too busy because many ZCTAs were too close to each other.  I decided what I really cared about was the ZCTAs with the largest percentage of Hispanics and the ZCTAs with the largest percentage of blacks:
  
  
<div class="separator" style="clear: both;"><a href="https://i1.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAeTA8RiM4CbFER5LHJDZiX1etA38Em3GklMums3-nb-kHGlQa7gMxYaTAxzAmN894-oPwInL0wHVdx0QAeZH88FTBCbiHNyGLOvdJ7vDbaA2cmFycWfsWjaclp-t03WbkAiSMxqD9TaovMRcTy1VCA13evDGO7apJTdc8cbzVgcvn4s_N9khny4B8vUY/s894/map4.png?ssl=1" style="display: block; padding: 1em 0; text-align: center; " rel="nofollow" target="_blank"><img alt="" border="0" width="450" data-original-height="438" data-original-width="450" src="https://i0.wp.com/blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAeTA8RiM4CbFER5LHJDZiX1etA38Em3GklMums3-nb-kHGlQa7gMxYaTAxzAmN894-oPwInL0wHVdx0QAeZH88FTBCbiHNyGLOvdJ7vDbaA2cmFycWfsWjaclp-t03WbkAiSMxqD9TaovMRcTy1VCA13evDGO7apJTdc8cbzVgcvn4s_N9khny4B8vUY/s600/map4.png?resize=450%2C438&#038;ssl=1" data-recalc-dims="1"/></a></div>
  
<h3>What&#8217;s the value in all this?</h3>
  
     
     What&#8217;s the value in all this?  Your pharmacy, department store, and video streaming service all use information about you to provide personalized recommendations based on what they think you&#8217;ll buy or watch next.  Do our political parties produce customized political messages for different neighborhoods based on what they think are their residents&#8217; major political issues?  Is race/ethnicity an important (although of course not the only) variable in people&#8217;s political issues?  I think maybe our political parties should think about making greater use of demographic and geographic information when deciding how to communicate with voters. <p>
  


<h3>R code</h3>
     
    There is a lot of R code.  I was considerably helped by Microsoft Copilot and Google Gemini.
    
  
<pre>

# Race by ZCTA
# MY_KEY &lt;- &quot;xxxxx&quot;
# Sys.setenv(CENSUS_API_KEY = &quot;MY_KEY&quot;)

library(tidycensus)
library(tigris)
library(dplyr)
library(tidyr)
library(stringr)
library(sf)
library(ggplot2)
library(scatterpie)
library(gridExtra)

options(tigris_use_cache = TRUE)

# ------------------------------------------------------------
# Variable Definitions
# ------------------------------------------------------------

# 2020 Census P2 (Place-Level)
vars_p2 &lt;- c(
  total     = &quot;P2_001N&quot;,
  hispanic  = &quot;P2_002N&quot;,
  white_nh  = &quot;P2_005N&quot;,
  black_nh  = &quot;P2_006N&quot;,
  aian_nh   = &quot;P2_007N&quot;,
  asian_nh  = &quot;P2_008N&quot;,
  nhpi_nh   = &quot;P2_009N&quot;,
  other_nh  = &quot;P2_010N&quot;,
  multi_nh  = &quot;P2_011N&quot;
)

# 2020 Census DHC ZCTA Variables
vars_p5 &lt;- c(
  total_pop = &quot;P5_001N&quot;,
  white_nh  = &quot;P5_003N&quot;,
  black_nh  = &quot;P5_004N&quot;,
  aian_nh   = &quot;P5_005N&quot;,
  asian_nh  = &quot;P5_006N&quot;,
  nhpi_nh   = &quot;P5_007N&quot;,
  other_nh  = &quot;P5_008N&quot;,
  multi_nh  = &quot;P5_009N&quot;,
  hispanic  = &quot;P5_010N&quot;
)

# ACS Variables (for individual ZCTA analysis)
vars_acs &lt;- c(
  total     = &quot;B03002_001&quot;,
  white_nh  = &quot;B03002_003&quot;,
  black_nh  = &quot;B03002_004&quot;,
  aian_nh   = &quot;B03002_005&quot;,
  asian_nh  = &quot;B03002_006&quot;,
  nhpi_nh   = &quot;B03002_007&quot;,
  other_nh  = &quot;B03002_008&quot;,
  multi_nh  = &quot;B03002_009&quot;,
  hispanic  = &quot;B03002_012&quot;
)

# ------------------------------------------------------------
# Geography
# ------------------------------------------------------------

places_fl &lt;- places(state = &quot;FL&quot;, year = 2020)

fort_myers_shape &lt;- places_fl %&gt;%
  filter(NAME == &quot;Fort Myers&quot;)

zctas_fl &lt;- zctas(
  year = 2020,
  cb = TRUE,
  starts_with = &quot;339&quot;
)

# ------------------------------------------------------------
# CITY OF FORT MYERS RACE TABLE
# ------------------------------------------------------------

race_eth_city &lt;- get_decennial(
  geography = &quot;place&quot;,
  state = &quot;FL&quot;,
  variables = vars_p2,
  year = 2020,
  sumfile = &quot;pl&quot;,
  output = &quot;wide&quot;
) %&gt;%
  filter(NAME == &quot;Fort Myers city, Florida&quot;)

race_city &lt;- race_eth_city %&gt;%
  transmute(
    white_nh,
    hispanic,
    black_nh,
    other =
      aian_nh +
      asian_nh +
      nhpi_nh +
      other_nh +
      multi_nh
  )

race_table_city &lt;- race_city %&gt;%
  pivot_longer(
    everything(),
    names_to = &quot;race&quot;,
    values_to = &quot;population&quot;
  ) %&gt;%
  mutate(
    pct = population / sum(population)
  )

print(race_table_city)

# ------------------------------------------------------------
# CITY OF FORT MYERS MAP
# ------------------------------------------------------------

ggplot(fort_myers_shape) +
  geom_sf(
    fill = &quot;lightblue&quot;,
    color = &quot;black&quot;
  ) +
  labs(
    title = &quot;City of Fort Myers (2020 Census Boundary)&quot;
  ) +
  theme_minimal() +
  theme(
    plot.title = element_text(face = &quot;bold&quot;),
    axis.text  = element_blank(),
    axis.ticks = element_blank()
  )

# ------------------------------------------------------------
# CITY PIE CHART MAP
# ------------------------------------------------------------

ctr &lt;- st_coordinates(
  st_centroid(
    st_union(fort_myers_shape)
  )
)

pie_df_city &lt;- cbind(
  data.frame(
    x = ctr[1],
    y = ctr[2]
  ),
  race_city
)

ggplot() +
  geom_sf(
    data = fort_myers_shape,
    fill = &quot;lightblue&quot;,
    color = &quot;black&quot;
  ) +
  geom_scatterpie(
    data = pie_df_city,
    aes(x = x, y = y, r = 0.03),
    cols = names(race_city)
  ) +
  scale_fill_manual(
    values = c(
      white_nh = &quot;lightgray&quot;,
      hispanic = &quot;orange&quot;,
      black_nh = &quot;steelblue&quot;,
      other = &quot;purple&quot;
    )
  ) +
  labs(
    title = &quot;Fort Myers City Racial Composition&quot;,
    fill = &quot;Race/Ethnicity&quot;
  ) +
  coord_sf() +
  theme(
    plot.title = element_text(face = &quot;bold&quot;),
    axis.title = element_blank(),
    axis.text = element_blank(),
    axis.ticks = element_blank(),
    legend.title = element_text(face = &quot;bold&quot;),
    legend.text = element_text(face = &quot;bold&quot;)
  )

# ------------------------------------------------------------
# GREATER FORT MYERS ZCTA MAP
# ------------------------------------------------------------

fort_myers_zctas &lt;- zctas_fl %&gt;%
  st_filter(fort_myers_shape,
            .predicate = st_intersects)

ggplot() +
  geom_sf(
    data = fort_myers_zctas,
    fill = &quot;lightblue&quot;,
    color = &quot;black&quot;,
    alpha = 0.6
  ) +
  geom_sf(
    data = fort_myers_shape,
    fill = NA,
    color = &quot;red&quot;,
    linewidth = 1.8,
    linetype = &quot;dashed&quot;
  ) +
  geom_sf_label(
    data = fort_myers_zctas,
    aes(label = ZCTA5CE20),
    size = 3,
    fontface = &quot;bold&quot;
  ) +
  labs(
    title = &quot;Greater Fort Myers Area by ZCTA&quot;,
    subtitle = &quot;Dashed red line indicates city limits&quot;, 
    x = &quot;&quot;,
    y = &quot;&quot;) + 
  theme_minimal() +
  theme(
    plot.title = element_text(face = &quot;bold&quot;),
    plot.subtitle = element_text(face = &quot;bold&quot;),
    axis.text = element_blank(),
    axis.ticks = element_blank()
  )

# ------------------------------------------------------------
# SINGLE ZCTA ANALYSIS (33905)
# ------------------------------------------------------------

race_eth_33905 &lt;- get_acs(
  geography = &quot;zcta&quot;,
  variables = vars_acs,
  year = 2020,
  output = &quot;wide&quot;
) %&gt;%
  filter(GEOID == &quot;33905&quot;)

race_33905 &lt;- race_eth_33905 %&gt;%
  transmute(
    white_nh = white_nhE,
    hispanic = hispanicE,
    black_nh = black_nhE,
    other =
      aian_nhE +
      asian_nhE +
      nhpi_nhE +
      other_nhE +
      multi_nhE
  )

race_table_33905 &lt;- race_33905 %&gt;%
  pivot_longer(
    everything(),
    names_to = &quot;race&quot;,
    values_to = &quot;population&quot;
  ) %&gt;%
  mutate(
    pct = population / sum(population)
  )

print(race_table_33905)

# ------------------------------------------------------------
# ZCTA 33905 PIE MAP
# ------------------------------------------------------------

zcta_33905 &lt;- zctas_fl %&gt;%
  filter(ZCTA5CE20 == &quot;33905&quot;)

coords_33905 &lt;- st_coordinates(
  st_centroid(zcta_33905)
)

pie_df_33905 &lt;- race_33905 %&gt;%
  mutate(
    x = coords_33905[1,1],
    y = coords_33905[1,2]
  )


ggplot() +
  geom_sf(
    data = zcta_33905,
    fill = &quot;lightblue&quot;,
    color = &quot;black&quot;
  ) +
  geom_scatterpie(
    data = pie_df_33905,
    aes(x = x, y = y, r = 0.025),
    cols = c(
      &quot;white_nh&quot;,
      &quot;hispanic&quot;,
      &quot;black_nh&quot;,
      &quot;other&quot;
    )
  ) +
  scale_fill_manual(
    values = c(
      white_nh = &quot;lightgray&quot;,
      hispanic = &quot;orange&quot;,
      black_nh = &quot;steelblue&quot;,
      other = &quot;purple&quot;
    )
  ) +
  labs(
    title = &quot;ZCTA 33905 Racial Composition&quot;,
    fill = &quot;Race/Ethnicity&quot;
  ) +
  coord_sf() +
  theme_minimal() +
  theme(
    plot.title = element_text(face = &quot;bold&quot;),
    axis.title = element_blank(),
    axis.text = element_blank(),
    axis.ticks = element_blank()
  )

# ------------------------------------------------------------
# ALL FORT MYERS AREA ZCTAS
# ------------------------------------------------------------

fort_myers_list &lt;- c(
  &quot;33901&quot;,&quot;33903&quot;,&quot;33905&quot;,&quot;33907&quot;,
  &quot;33908&quot;,&quot;33912&quot;,&quot;33913&quot;,&quot;33916&quot;,
  &quot;33917&quot;,&quot;33919&quot;,&quot;33931&quot;,&quot;33965&quot;,
  &quot;33966&quot;,&quot;33967&quot;
)

fort_myers_master &lt;- get_decennial(
  geography = &quot;zcta&quot;,
  variables = vars_p5,
  year = 2020,
  sumfile = &quot;dhc&quot;,
  output = &quot;wide&quot;,
  geometry = TRUE
)

fort_myers_projected &lt;- fort_myers_master %&gt;%
  filter(GEOID %in% fort_myers_list) %&gt;%
  mutate(
    zcta = str_extract(NAME, &quot;\\d{5}&quot;),
    pct_hispanic =
      100 * hispanic / total_pop,
    pct_black_nh =
      100 * black_nh / total_pop,
    minority_other_nh =
      aian_nh +
      asian_nh +
      nhpi_nh +
      other_nh +
      multi_nh
  ) %&gt;%
  st_transform(2237)

# ------------------------------------------------------------
# POPULATION MATRIX
# ------------------------------------------------------------

fort_myers_population_table &lt;- fort_myers_projected %&gt;%
  st_drop_geometry() %&gt;%
  select(
    zcta,
    total_pop,
    white_nh,
    hispanic,
    black_nh,
    minority_other_nh
  ) %&gt;%
  arrange(zcta)

print(fort_myers_population_table)

# ------------------------------------------------------------
# CENTROIDS
# ------------------------------------------------------------

centroids_master &lt;- fort_myers_projected %&gt;%
  st_centroid() %&gt;%
  mutate(
    X = st_coordinates(.)[,1],
    Y = st_coordinates(.)[,2]
  ) %&gt;%
  st_drop_geometry()

top_5_hispanic &lt;- centroids_master %&gt;%
  slice_max(
    pct_hispanic,
    n = 5
  ) %&gt;%
  mutate(
    label_text =
      paste0(
        zcta,
        &quot;\n&quot;,
        round(pct_hispanic, 1),
        &quot;%&quot;
      )
  )

top_5_black &lt;- centroids_master %&gt;%
  slice_max(
    pct_black_nh,
    n = 5
  ) %&gt;%
  mutate(
    label_text =
      paste0(
        zcta,
        &quot;\n&quot;,
        round(pct_black_nh, 1),
        &quot;%&quot;
      )
  )

# ------------------------------------------------------------
# BASE MAP
# ------------------------------------------------------------

base_map &lt;- ggplot() +
  geom_sf(
    data = fort_myers_projected,
    fill = &quot;lightblue&quot;,
    color = &quot;gray60&quot;
  ) +
  theme_minimal() +
  theme(
    plot.title = element_text(
      face = &quot;bold&quot;,
      hjust = .5
    ),
    axis.title = element_blank(),
    axis.text = element_blank(),
    panel.grid = element_blank()
  ) +
  coord_sf(datum = NA)

# ------------------------------------------------------------
# TOP 5 HISPANIC
# ------------------------------------------------------------

map_hispanic &lt;- base_map +
  geom_text(
    data = top_5_hispanic,
    aes(
      X,
      Y,
      label = label_text
    ),
    fontface = &quot;bold&quot;
  ) +
  labs(
    title = &quot;Top 5 ZCTAs by Hispanic %&quot;
  )

# ------------------------------------------------------------
# TOP 5 BLACK NH
# ------------------------------------------------------------

map_black &lt;- base_map +
  geom_text(
    data = top_5_black,
    aes(
      X,
      Y,
      label = label_text
    ),
    fontface = &quot;bold&quot;
  ) +
  labs(
    title = &quot;Top 5 ZCTAs by Black (NH) %&quot;
  )

# ------------------------------------------------------------
# DISPLAY FINAL MAPS
# ------------------------------------------------------------

grid.arrange(
  map_hispanic,
  map_black,
  ncol = 2
)

</pre><p>
  
End
</font>
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://onlinecollegemathteacher.blogspot.com/2026/11/distribution-of-race-by-zcta.html"> Online College Math Teacher</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/distribution-of-race-by-zcta/">Distribution of race by ZCTA</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404810</post-id>	</item>
		<item>
		<title>The rule of law and economic growth: what three decades of data show</title>
		<link>https://www.r-bloggers.com/2026/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/</link>
		
		<dc:creator><![CDATA[Giles]]></dc:creator>
		<pubDate>Wed, 11 Nov 2026 04:48:36 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://www.gilesd-j.com/?p=4806</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; "> Weighted by population, the rule of law has slipped globally since 1996. India and the United States contributed a large part to this decline, while China's net impact was negligible thanks to a recovery from the mid 00's. Countries with a stronger rule of law tend to be wealthier, and those ...</div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/">The rule of law and economic growth: what three decades of data show</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/"> Data Analytics and AI Archives - Giles</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>

<p class="wp-block-paragraph"><strong>TLDR:</strong> <em>Weighted by population, the rule of law has slipped globally since 1996. India and the United States contributed a large part to this decline, while China’s net impact was negligible thanks to a recovery from the mid 00’s. Countries with a stronger rule of law tend to be wealthier, and those where it improved were more likely to grow faster, but there are plenty of exceptions to the rule, with China experiencing higher growth than might be expected from changes in its rule of law score, alongside Armenia, Bosnia and Herzegovina, Guyana and Myanmar.</em></p>



<p class="wp-block-paragraph"><strong>How AI was used:</strong> AI was used to write the first draft of analysis included in this post. I then collaborated with AI to refine the post until it resembled what I had in mind: the majority of writing is therefore my own. AI also was used to put the post in a format compatible with WordPress. </p>



<h2 class="wp-block-heading">Background</h2>



<p class="wp-block-paragraph">One of the reasons consulting suits me is that it provides me with an <em>almost</em> unlimited number of rabbit holes to get lost in. One such example that came up in 2025, was a project exploring the links between economic growth and the rule of law for the <a href="https://binghamcentre.biicl.org/" rel="nofollow" target="_blank">Bingham Centre for the Rule of Law</a> and the <a href="https://www.lawsociety.org.uk/" rel="nofollow" target="_blank">Law Society of England and Wales.</a> If you’re interested in exploring this further, I’d recommend <a href="https://binghamcentre.biicl.org/publications/the-rule-of-law-and-the-institutional-roots-of-economic-performance" rel="nofollow" target="_blank">Dr Lopez-Gomez’s literature review</a>, which was a key outcome of the project.</p>



<p class="wp-block-paragraph">This is the fourth post in my series on the rule of law (see earlier posts: <a href="https://www.gilesd-j.com/2026/07/07/close-enough-using-the-wgi-as-a-proxy-for-the-wjp-rule-of-law-index/" rel="nofollow" target="_blank">1</a>, <a href="https://www.gilesd-j.com/2026/07/21/level-with-me-can-the-wgi-predict-the-wjps-rule-of-law-scores/" rel="nofollow" target="_blank">2</a> and <a href="https://www.gilesd-j.com/2026/08/10/institutions-are-a-package-deal-what-a-correlation-cant-tell-you-about-the-rule-of-law/" rel="nofollow" target="_blank">3</a>), with previous posts intended to demonstrate themes that this and future posts rely on, including that:</p>



<ul class="wp-block-list">
<li class="">The Worldwide Governance Indicator’s (WGI) Rule of Law index provides a reliable proxy for the World Justice Project’s (WJP) measure (with greater coverage over a longer period); and</li>



<li class="">It’s probably <a href="https://www.atlanticcouncil.org/in-depth-research-reports/issue-brief/why-the-rule-of-law-is-the-key-to-prosperity-lessons-from-thirty-years-of-data/" rel="nofollow" target="_blank">not a good idea</a> to solely use pairwise relationships for exploring how the rule of law interacts with other outcomes as these things are hard to measure, co-determined and <em>probably</em> operate within an interdependent system of institutions.</li>
</ul>



<h2 class="wp-block-heading">Four themes</h2>



<p class="wp-block-paragraph">Although I had grander plans when I first started drafting this post, I’ve scaled back my ambition to focus on four themes that provide useful context for future posts:</p>



<ul class="wp-block-list">
<li class="">Globally, the rule of law has deteriorated since the mid 90s.</li>



<li class="">Countries with stronger rule of law are wealthier.</li>



<li class="">Countries that improve their rule of law tend to grow faster.</li>



<li class="">But, the rule of law isn’t a <em>necessary</em> or <em>sufficient</em> condition for economic growth.</li>
</ul>



<h2 class="wp-block-heading">Here be dragons</h2>



<p class="wp-block-paragraph">To avoid the need for weasel words and vague descriptors it’s worth remembering that cross-country analysis is a minefield:</p>



<ul class="wp-block-list">
<li class=""><strong>Institutions like the rule of law (etc) are hard to define and measure:</strong> Institutions are the rules, norms and conventions that shape, constrain and guide human behaviour and interactions. Institutions can run from the grand rules and expectations that ensure peaceful democratic transitions after an election to the mundane social norm that ensures Bill from Accounting doesn’t microwave his tuna salad at lunch. Whether the former is more important than the latter is a matter of taste, but being all encompassing, interdependent and hard to define also makes institutions difficult to measure. This is particularly true for perception-based measures as someone asked how well the rule of law works is likely to be thinking about democracy, accountability and government effectiveness too, while potentially making qualitative judgements that lean towards accepted Western models of governance.</li>



<li class=""><strong>Interesting things tend to be correlated with one another:</strong> As institutions like the <em>rule of law</em> operate within an interdependent system (<a href="https://www.gilesd-j.com/2026/08/10/institutions-are-a-package-deal-what-a-correlation-cant-tell-you-about-the-rule-of-law/" rel="nofollow" target="_blank">see last post</a>), everything tends to be correlated with one another whether or not they are causally related. Economic growth probably depends on the quality of a country’s legal system, but the legal system itself likely also depends on economic growth too as it influences the resourcing available for government to invest in it. This makes it difficult to talk about the <em>rule of law</em> as a discrete characteristic independent of other institutions.</li>



<li class=""><strong>National statistics obscure the relationships that matter:</strong> Cross-country statistics are not always representative of specific actors or the underlying causal mechanisms that we’re interested in. For instance, <a href="https://arxiv.org/abs/2603.16007" rel="nofollow" target="_blank">economic and social relationships sometimes transcend the nation-state</a> (the unit of focus for cross-country analysis).</li>



<li class=""><strong>Institutions are sticky and slow moving:</strong> We consider a time series of data running from 1996 to 2023, which is just short of three decades. Although this might feel long-run enough, leading studies in the field cover much longer periods e.g. 100+ years (see <a href="https://www.aeaweb.org/articles?id=10.1257%2Faer.91.5.1369" rel="nofollow" target="_blank">1</a> and <a href="https://cepr.org/publications/dp20214" rel="nofollow" target="_blank">2</a>). In practical terms, this might mean that the strength and direction of relationships examined here might not generalise over longer periods.</li>
</ul>



<p class="wp-block-paragraph">But, I haven’t picked the themes covered here by accident: each of them tells a story supported by the literature, which I’ve done my best to cite throughout.</p>



<h2 class="wp-block-heading">Project setup</h2>



<h3 class="wp-block-heading"><strong>Data</strong></h3>



<p class="wp-block-paragraph">The rule of law measure is the <em>Rule of Law</em> dimension of the World Bank’s WGI, 2025 release, which covers 1996 to 2024.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn1" class="footnote-ref" id="fnref1" role="doc-noteref" rel="nofollow" target="_blank"><sup>1</sup></a> The WGI’s RoL attempts to measure the quality of contract enforcement, property rights, the police, the courts and the likelihood of crime and violence. The WGI’s measure has been selected on the basis of it covering a larger number of countries over a longer period. Earlier posts in the series also indicated it provides a reliable proxy for the WJP’s more comprehensive RoL measure. The data can be downloaded <a href="https://www.gilesd-j.com/shared_resources/blogs/260310_RoL/wgidataset_with_sourcedata-2025.xlsx" rel="nofollow" target="_blank">here</a>.</p>



<p class="wp-block-paragraph">Statistical measures, namely GDP per capita (PPP), GDP per capita growth, population and oil and gas rents, along with each country’s World Bank region, have been drawn from the World Development Indicators via the wbstats package. They were downloaded once, on 22 October 2026, and saved, so the results don’t change when the World Bank revises its figures. A snapshot of the data used here can be downloaded <a href="https://www.gilesd-j.com/shared_resources/blogs/260310_RoL/261022%20-%20wdi_indicators.csv" rel="nofollow" target="_blank">here</a>.</p>



<p class="wp-block-paragraph">Country income classifications were the earliest classifications available from the World Bank’s historical classification data (<a href="https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups" rel="nofollow" target="_blank">available here</a>).</p>



<p class="wp-block-paragraph">Pacific island small states are the World Bank’s group of that name, taken from its current country classification file (CLASS.xlsx, downloaded on 7 November 2026 from the <a href="https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups" rel="nofollow" target="_blank">same page</a>).</p>



<p class="wp-block-paragraph">Access to the sea, as the share of each country’s population living within 100 km of a coast or a navigable river, and the malaria index came from the geography dataset of Gallup, Sachs and Mellinger,<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn2" class="footnote-ref" id="fnref2" role="doc-noteref" rel="nofollow" target="_blank"><sup>2</sup></a> downloaded on 7 November 2026 from <a href="https://ideas.repec.org/p/boc/bocins/geodata.html" rel="nofollow" target="_blank">RePEc</a>.</p>



<p class="wp-block-paragraph">The four code chunks below are used for loading libraries, specifying key assumptions, and importing and joining the data.</p>



<p class="wp-block-paragraph"><strong>Code:</strong> setup</p>



<p class="wp-block-paragraph"><em>The code below loads the libraries, sets key assumptions and defines standard chart styles and labels for the rest of the post.</em></p>



<pre># load the packages we'll need
library(tidyverse)
library(readxl)
library(janitor)
library(ggrepel)

#file paths
ref_path &lt;- list(
  wgi            = file.path(&quot;Data&quot;, &quot;wgidataset_with_sourcedata-2025.xlsx&quot;),
  income_history = file.path(&quot;Data&quot;, &quot;OGHIST_2026_10_08.xlsx&quot;),  # World Bank historical income groups
  wdi            = file.path(&quot;Data&quot;, &quot;261022 - wdi_indicators.csv&quot;),  # WDI indicators and regions
  pacific        = file.path(&quot;Data&quot;, &quot;261107 - pacific_island_small_states.csv&quot;)  # World Bank Pacific island small states
)

#years covered
ref_year &lt;- list(
  start        = 1996,      # the first WGI year
  end          = 2023,      # the last year with near-complete WDI coverage for most series
  rents_window = 2019:2023  # the last five years, averaged for oil and gas rents, which swing with prices
)

#cut-offs that define samples and groups
ref_cutoff &lt;- list(
  min_growth_years         = 20,  # countries with fewer years of growth data are dropped
  large_producer_rents_pct = 10,  # oil and gas rents above this share of GDP mark a large producer
  min_group_countries      = 5,   # region and income groups with fewer countries are left off the correlation chart
  outlier_sd               = 2    # countries further than this many standard deviations above the line are named on the growth chart
)

#WDI indicators: short name = World Bank code
ref_wdi_indicators &lt;- c(
  econ_gdppc_ppp        = &quot;NY.GDP.PCAP.PP.KD&quot;,   # GDP per capita, PPP, constant international $
  econ_gdppc_growth_pct = &quot;NY.GDP.PCAP.KD.ZG&quot;,   # GDP per capita growth, annual %
  pop_tot_nmb           = &quot;SP.POP.TOTL&quot;,         # population
  res_oil_rents_pct     = &quot;NY.GDP.PETR.RT.ZS&quot;,   # oil rents, % of GDP
  res_gas_rents_pct     = &quot;NY.GDP.NGAS.RT.ZS&quot;    # natural gas rents, % of GDP
)

#colours (from my site palette), set once for every chart
ref_col &lt;- list(
  navy       = &quot;#0C1446&quot;,  # titles and strip labels
  charcoal   = &quot;#2B2E45&quot;,  # body text
  grey       = &quot;#73768C&quot;,  # axis labels, captions, annotations
  grey_light = &quot;#C7CBD8&quot;,  # background points and context
  grey_pale  = &quot;#E2E8F0&quot;,  # gridlines and zero lines
  blue       = &quot;#2740DA&quot;,  # the series the chart is about
  red        = &quot;#922C40&quot;,  # decline
  orange     = &quot;#DC9750&quot;,  # a second group shown beside the blue one
  green      = &quot;#16AF8E&quot;   # improvement
)
#red and green always mean decline and improvement; the four hues in order for charts with several groups
ref_col$direction &lt;- c(Decline = ref_col$red, Improvement = ref_col$green)
ref_col$series    &lt;- c(ref_col$blue, ref_col$red, ref_col$orange, ref_col$green)

#chart text: the source caption, how to read a WGI axis, and the large producer legend
# WGI scores run from about -2.5 to +2.5, centred on the average country.
# Arrows turn with the text on a vertical axis, so vertical axes get words instead.
ref_lab &lt;- list(
  caption             = &quot;Source: Worldwide Governance Indicators (2025); World Development Indicators.&quot;,
  axis_score          = &quot;← weaker      0 = average country      stronger →&quot;,
  axis_score_vertical = &quot;higher = stronger; 0 = average country&quot;,
  axis_change         = &quot;← worsened      0 = no change      improved →&quot;,
  large_producers     = paste0(&quot;Oil and gas rents above &quot;, ref_cutoff$large_producer_rents_pct, &quot;% of GDP&quot;),
  clear_of_error      = &quot;* The change is larger than its 90% measurement band, which lies wholly on one side of zero.&quot;,
  income_start        = &quot;Income groups are each country's earliest World Bank classification (for most, based on 1987 income), not its current one.&quot;
)

#series colours for charts with up to four groups
options(ggplot2.discrete.colour = ref_col$series,
        ggplot2.discrete.fill   = ref_col$series)

# Open Sans needs to be installed; ragg (set in the YAML) picks it up.
theme_set(
  theme_minimal(base_size = 11, base_family = &quot;Open Sans&quot;) +
    theme(text                  = element_text(colour = ref_col$charcoal),
          axis.text             = element_text(colour = ref_col$grey),
          axis.title            = element_text(size = rel(0.9)),
          axis.title.x          = element_text(margin = margin(t = 8)),
          axis.title.y          = element_text(margin = margin(r = 8)),
          panel.grid.minor      = element_blank(),
          panel.grid.major.x    = element_blank(),
          panel.grid.major.y    = element_line(colour = ref_col$grey_pale, linewidth = 0.4),
          strip.text            = element_text(colour = ref_col$navy, face = &quot;bold&quot;, hjust = 0),
          legend.position       = &quot;top&quot;,
          legend.justification  = &quot;left&quot;,
          legend.title          = element_blank(),
          plot.title            = element_text(colour = ref_col$navy, face = &quot;bold&quot;,
                                               size = rel(1.15)),
          plot.subtitle         = element_text(colour = ref_col$grey,
                                               margin = margin(b = 10)),
          plot.title.position   = &quot;plot&quot;,
          plot.caption         = element_text(colour = ref_col$grey, face = &quot;italic&quot;,
                                               size = rel(0.75), hjust = 0),
          plot.caption.position = &quot;plot&quot;,
          plot.margin           = margin(10, 15, 10, 10))
)
update_geom_defaults(&quot;text&quot;, list(family = &quot;Open Sans&quot;, colour = ref_col$charcoal))</pre>



<h2 class="wp-block-heading">Importing data</h2>



<p class="wp-block-paragraph"><strong>Code:</strong> import-wgi</p>



<p class="wp-block-paragraph"><em>The code below imports the WGI and recodes countries so they align with the World Bank data.</em></p>



<pre># Read the Rule of Law sheet. The sheet has one unnamed column; &quot;unique_quiet&quot;
# gives it a name without printing a message.
dta_wgi_rol &lt;- read_excel(ref_path$wgi, sheet = &quot;rl&quot;, .name_repair = &quot;unique_quiet&quot;) |&gt;
  clean_names()

# Keep the columns we need, with shorter names. Missing estimates may be
# stored as text (such as &quot;..&quot;), so the scores are converted to numbers.
dta_wgi_rol &lt;- dta_wgi_rol |&gt;
  mutate(year       = as.integer(year),
         wgi_rol    = as.numeric(governance_estimate_approx_2_5_to_2_5),
         wgi_rol_se = as.numeric(standard_error_estimate)) |&gt;
  select(iso3c = economy_code, country = economy_name, year, wgi_rol, wgi_rol_se)

# Older WGI releases used a few codes that are not ISO codes. This lookup
# gives the ISO code for each one.
lkp_iso_fixes &lt;- c(ROM = &quot;ROU&quot;, ZAR = &quot;COD&quot;, TMP = &quot;TLS&quot;,
                   ADO = &quot;AND&quot;, KSV = &quot;XKX&quot;, WBG = &quot;PSE&quot;)

# Swap in the ISO code where the lookup has one; otherwise keep the code as it is
dta_wgi_rol &lt;- dta_wgi_rol |&gt;
  mutate(iso3c = coalesce(unname(lkp_iso_fixes[iso3c]), iso3c))</pre>



<p class="wp-block-paragraph"><strong>Code:</strong> import-wdi</p>



<p class="wp-block-paragraph"><em>The code below imports the WDI data and the World Bank’s historical income classifications.</em></p>



<p class="wp-block-paragraph"><em>Each country is grouped by its earliest World Bank income classification to avoid the groups directly mirroring rule of law scores i.e. due to simultaneity between economic growth and the rule of law. This isn’t a perfect solution, but will have to suffice in the absence of <a href="https://en.wikipedia.org/wiki/Instrumental_variables_estimation" rel="nofollow" target="_blank">an instrument</a>.</em></p>



<pre># WDI indicators and World Bank regions, downloaded on 22 October 2026 with
# wbstats::wb_data(indicator = ref_wdi_indicators, start_date = 1996, end_date = 2023),
# and each country's region from wbstats::wb_countries(), with aggregates such as
# &quot;World&quot; dropped
dta_wdi &lt;- read_csv(ref_path$wdi, show_col_types = FALSE) |&gt;
  select(iso3c, country, region, year, all_of(names(ref_wdi_indicators))) |&gt;
  mutate(year = as.integer(year))

# Income group names: World Bank code = full name, and full name = short name for charts
lkp_income &lt;- list(
  full  = c(L  = &quot;Low income&quot;,          LM = &quot;Lower middle income&quot;,
            UM = &quot;Upper middle income&quot;, H  = &quot;High income&quot;),
  short = c(&quot;Low income&quot;          = &quot;Low&quot;,
            &quot;Lower middle income&quot; = &quot;Lower middle&quot;,
            &quot;Upper middle income&quot; = &quot;Upper middle&quot;,
            &quot;High income&quot;         = &quot;High&quot;)
)

# The World Bank's historical income classifications. The fiscal years (FY89
# to FY27) are in row 5, so the four title rows above it are skipped. Every
# column is read as text, because the codes are letters. The first column
# holds the country code.
lkp_income_start &lt;- read_excel(ref_path$income_history, sheet = &quot;Country Analytical History&quot;,
                               skip = 4, col_types = &quot;text&quot;, .name_repair = &quot;unique_quiet&quot;) |&gt;
  rename(iso3c = 1)

# Keep the country rows only. This drops the row of calendar years under the
# fiscal years, and the notes at the foot of the sheet.
lkp_income_start &lt;- lkp_income_start |&gt;
  filter(str_detect(iso3c, &quot;^[A-Z]{3}$&quot;))

# Reshape to one line per country and fiscal year
lkp_income_start &lt;- lkp_income_start |&gt;
  pivot_longer(starts_with(&quot;FY&quot;), names_to = &quot;fiscal_year&quot;, values_to = &quot;income_code&quot;)

# Keep the years in which a country was classified. &quot;..&quot; means not classified
# that year. &quot;LM*&quot; is Yemen with a footnote, so the asterisk is removed first.
lkp_income_start &lt;- lkp_income_start |&gt;
  mutate(income_code = str_remove(income_code, &quot;\\*&quot;)) |&gt;
  filter(income_code %in% names(lkp_income$full))

# Turn the fiscal year into a number: FY89 to FY99 are 1989 to 1999, and FY00
# onwards are 2000 onwards
lkp_income_start &lt;- lkp_income_start |&gt;
  mutate(fiscal_year = as.integer(str_remove(fiscal_year, &quot;FY&quot;)),
         fiscal_year = if_else(fiscal_year &gt;= 80, 1900L + fiscal_year, 2000L + fiscal_year))

# Keep each country's earliest classification: slice_min() keeps the line with
# the smallest fiscal year for each country
lkp_income_start &lt;- lkp_income_start |&gt;
  slice_min(fiscal_year, n = 1, with_ties = FALSE, by = iso3c)

# The columns used in the rest of the post: the income group's full name, the
# fiscal year it comes from, and a simpler split of high income against
# everyone else, used in later charts
lkp_income_start &lt;- lkp_income_start |&gt;
  mutate(econ_income_start    = unname(lkp_income$full[income_code]),
         econ_income_start_fy = fiscal_year,
         econ_hic_start       = if_else(econ_income_start == &quot;High income&quot;,
                                        &quot;High income&quot;, &quot;Low & middle income&quot;)) |&gt;
  select(iso3c, econ_income_start, econ_income_start_fy, econ_hic_start)

# Number of countries in each starting income group
count(lkp_income_start, econ_income_start, econ_hic_start)
# A tibble: 4 × 3
  econ_income_start   econ_hic_start          n
  &lt;chr&gt;               &lt;chr&gt;               &lt;int&gt;
1 High income         High income            54
2 Low income          Low & middle income    51
3 Lower middle income Low & middle income    78
4 Upper middle income Low & middle income    40</pre>



<p class="wp-block-paragraph"><strong>Code:</strong> join</p>



<p class="wp-block-paragraph"><em>The code below joins WGI and WDI datasets into a single panel dataset. Countries that failed to join are listed, as are countries scored in 1996 whose first income classification came later.</em></p>



<pre># Add the WGI scores to the WDI data, matching on country code and year. The
# WDI data already has the country names, so the WGI's are dropped.
dta_panel &lt;- dta_wdi |&gt;
  left_join(select(dta_wgi_rol, -country), by = join_by(iso3c, year), relationship = &quot;one-to-one&quot;)

# Add each country's starting income group
dta_panel &lt;- dta_panel |&gt;
  left_join(lkp_income_start, by = join_by(iso3c), relationship = &quot;many-to-one&quot;)

# WGI economies with no match in the WDI data. Most are territories, such as
# Jersey, or economies the World Bank doesn't cover, such as Taiwan.
chk_wgi_unmatched &lt;- dta_wgi_rol |&gt;
  distinct(iso3c, country) |&gt;
  anti_join(distinct(dta_panel, iso3c), by = join_by(iso3c))

chk_wgi_unmatched
# A tibble: 9 × 2
  iso3c country                      
  &lt;chr&gt; &lt;chr&gt;                        
1 GUF   French Guiana                
2 MTQ   Martinique                   
3 TWN   Taiwan, China                
4 AIA   Anguilla                     
5 ANT   Netherlands Antilles (former)
6 REU   Réunion                      
7 COK   Cook Islands                 
8 NIU   Niue                         
9 JEY   Jersey, Channel Islands      
# Countries scored in 1996 whose first income classification came after 1996
chk_income_late &lt;- dta_panel |&gt;
  filter(year == ref_year$start, !is.na(wgi_rol)) |&gt;
  filter(is.na(econ_income_start) | econ_income_start_fy &gt; ref_year$start) |&gt;
  select(iso3c, country, econ_income_start, econ_income_start_fy)

chk_income_late
# A tibble: 4 × 4
  iso3c country econ_income_start   econ_income_start_fy
  &lt;chr&gt; &lt;chr&gt;   &lt;chr&gt;                              &lt;int&gt;
1 NRU   Naoero  High income                         2017
2 PLW   Palau   Upper middle income                 1998
3 SRB   Serbia  Upper middle income                 2008
4 TUV   Tuvalu  Lower middle income                 2011</pre>



<h2 class="wp-block-heading">Global rule of law trends</h2>



<p class="wp-block-paragraph">To get a sense how the rule of law has changed, the code block below calculates the average rule of law score since 1996 across countries. Both unweighted and population-weighted scores are calculated, with the latter indicating how the rule of law has changed for the <em>average person</em>.</p>



<p class="wp-block-paragraph"><strong>Note:</strong> Revisions to the WGI’s methodology in 2025 improve the ability for scores to be compared over time (see the World Bank’s methodology note, cited in the Data section above).</p>



<p class="wp-block-paragraph"><strong>Code:</strong> population-weighted-trends</p>



<p class="wp-block-paragraph"><em>The code below calculates the global unweighted and weighted rule of law score. Only countries with WGI scores in the first year are included so the number of countries remain fixed.</em></p>



<pre># The countries with a score in 1996. Only these countries are kept, so the
# same countries are averaged in every year.
ref_countries_1996 &lt;- dta_panel |&gt;
  filter(year == ref_year$start, !is.na(wgi_rol)) |&gt;
  pull(iso3c)

# Keep those countries, in every year where they have a score, a standard
# error, a population and a starting income group
dta_panel_1996 &lt;- dta_panel |&gt;
  filter(iso3c %in% ref_countries_1996,
         !is.na(wgi_rol),
         !is.na(wgi_rol_se),
         !is.na(pop_tot_nmb),
         !is.na(econ_income_start))

# Order the income groups from poorest to richest (by the median GDP per
# capita of each group), so charts list them in that order
dta_panel_1996 &lt;- dta_panel_1996 |&gt;
  mutate(econ_income_start = fct_reorder(econ_income_start, econ_gdppc_ppp,
                                         .fun = median, .na_rm = TRUE))

# The world average in each year, weighted by population and unweighted.
# Each standard error combines the WGI's country standard errors using the
# same weights as the average. This assumes the measurement errors of
# different countries are independent of one another.
sum_rol_world_avg &lt;- dta_panel_1996 |&gt;
  summarise(
    wgi_rol_popw    = weighted.mean(wgi_rol, w = pop_tot_nmb),
    wgi_rol_unw     = mean(wgi_rol),
    wgi_rol_popw_se = sqrt(sum((pop_tot_nmb / sum(pop_tot_nmb))^2 * wgi_rol_se^2)),
    wgi_rol_unw_se  = sqrt(sum(wgi_rol_se^2)) / n(),
    nmb_countries   = n(),
    .by = year) |&gt;
  arrange(year)

# Change in the weighted average since 1996
sum_rol_world_avg &lt;- sum_rol_world_avg |&gt;
  mutate(wgi_rol_popw_change = wgi_rol_popw - wgi_rol_popw[year == ref_year$start])

# The change from 1996 to 2023, quoted in the text below
sum_rol_world_change &lt;- sum_rol_world_avg$wgi_rol_popw_change[sum_rol_world_avg$year == ref_year$end]</pre>



<h2 class="wp-block-heading"><em>The rule of law has declined globally</em></h2>



<p class="wp-block-paragraph">The chart below presents weighted and unweighted rule of law scores since 1996. Unweighted scores can be thought of as how the rule of law has changed for the average <em>country</em>, while the weighted scores are intended to show how it has changed for the average <em>person.</em> In both cases the rule of law appears to have declined globally, although the declines in unweighted scores only just clear the measurement error bands.</p>



<p class="wp-block-paragraph"><strong>Note:</strong> Shaded bands are based on the standard errors provided by the WGI and can be thought of as margins of error. In their note on analytical issues with the WGI, Kaufmann, Kraay and Mastruzzi suggest that overlapping intervals can be thought of as a conservative test for whether two scores are the same.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn3" class="footnote-ref" id="fnref3" role="doc-noteref" rel="nofollow" target="_blank"><sup>3</sup></a> And while the generalized declines might be cause for alarm, the population-weighted fall of 0.22 is smaller than the 0.31 difference in scores between Denmark and New Zealand in 2023. This doesn’t mean that the changes aren’t <em>significant</em> from a governance standpoint, but just that the scale of these declines doesn’t necessarily point to the world becoming lawless.</p>



<p class="wp-block-paragraph"><strong>Figure: World Rule of Law, population-weighted and unweighted, 1996–2023</strong></p>



<p class="wp-block-paragraph"><strong>Code:</strong> plot-world-trend</p>



<p class="wp-block-paragraph"><em>The code below produces a plot with the weighted and unweighted rule of law score since 1996. The shaded bands are based on 90 percent margins of errors provided by the WGI. The contributions to the average world score change is also estimated in</em> dta_country_contrib<em>, and the five most populous countries are picked out for the text and charts below.</em></p>



<pre># Stack the two averages into one table, one under the other, so each can be
# drawn as its own line
sum_plt_rol_world &lt;- bind_rows(
  sum_rol_world_avg |&gt;
    select(year, wgi_rol = wgi_rol_popw, wgi_rol_se = wgi_rol_popw_se) |&gt;
    mutate(measure = &quot;Population-weighted&quot;),
  sum_rol_world_avg |&gt;
    select(year, wgi_rol = wgi_rol_unw, wgi_rol_se = wgi_rol_unw_se) |&gt;
    mutate(measure = &quot;Unweighted&quot;)
)

plt_rol_world_trend &lt;- sum_plt_rol_world |&gt;
  ggplot(aes(x = year, y = wgi_rol, colour = measure, fill = measure)) +
  geom_hline(yintercept = 0, colour = ref_col$grey_pale) +
  # 90% band: 1.645 standard errors either side of the average
  geom_ribbon(aes(ymin = wgi_rol - 1.645 * wgi_rol_se,
                  ymax = wgi_rol + 1.645 * wgi_rol_se),
              colour = NA, alpha = 0.15) +
  geom_line(linewidth = 0.9) +
  geom_text(data = filter(sum_plt_rol_world, year == max(year)),
            aes(label = measure), hjust = 0, nudge_x = 0.6, size = 3.2) +
  scale_x_continuous(expand = expansion(mult = c(0.02, 0.22))) +
  theme(legend.position = &quot;none&quot;) +
  labs(title    = &quot;Weighted by population, the world's Rule of Law score has slipped since 1996&quot;,
       subtitle = paste0(&quot;Average of the &quot;, sum_rol_world_avg$nmb_countries[1],
                         &quot; countries scored in 1996, with 90% measurement bands&quot;),
       x = NULL, y = paste0(&quot;Rule of Law estimate (WGI units)\n&quot;, ref_lab$axis_score_vertical),
       caption = ref_lab$caption)

plt_rol_world_trend</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i2.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-plot-world-trend-1.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<pre># How much each country added to, or took away from, the change in the
# weighted world average. The weighted average in a year is the sum, across
# countries, of population share x score. So a country's contribution to the
# change is its share x score in 2023 less its share x score in 1996, and the
# contributions add up to the change in the world line exactly.

# Step 1: each country's share of the total population in 1996 and in 2023
dta_country_contrib &lt;- dta_panel_1996 |&gt;
  filter(year %in% c(ref_year$start, ref_year$end)) |&gt;
  mutate(pop_share = pop_tot_nmb / sum(pop_tot_nmb), .by = year) |&gt;
  mutate(period = if_else(year == ref_year$start, &quot;start&quot;, &quot;end&quot;)) |&gt;
  select(iso3c, country, period, pop_share, wgi_rol, wgi_rol_se)

# Step 2: put the two years side by side, one line per country, with the
# columns pop_share_start, pop_share_end, wgi_rol_start, wgi_rol_end,
# wgi_rol_se_start and wgi_rol_se_end
dta_country_contrib &lt;- dta_country_contrib |&gt;
  pivot_wider(names_from = period, values_from = c(pop_share, wgi_rol, wgi_rol_se))

# Step 3: each country's contribution, and its change in score
dta_country_contrib &lt;- dta_country_contrib |&gt;
  mutate(contrib        = pop_share_end * wgi_rol_end - pop_share_start * wgi_rol_start,
         wgi_rol_change = wgi_rol_end - wgi_rol_start)

# The five most populous countries in 2023, quoted in the text below and
# charted in the next chunk
dta_most_populous &lt;- dta_country_contrib |&gt;
  slice_max(pop_share_end, n = 5)

# Is each change clear of measurement error? The standard error of the change
# combines the 1996 and 2023 errors, treating them as independent, as for the
# region and income groups below. A change is clear when its whole 90% band
# (1.645 standard errors either side) lies on one side of zero.
dta_most_populous &lt;- dta_most_populous |&gt;
  mutate(wgi_rol_change_se = sqrt(wgi_rol_se_start^2 + wgi_rol_se_end^2),
         clear_of_error    = abs(wgi_rol_change) &gt; 1.645 * wgi_rol_change_se)

dta_most_populous &lt;- dta_most_populous |&gt;
  mutate(direction     = if_else(wgi_rol_change &lt; 0, &quot;Decline&quot;, &quot;Improvement&quot;),
         # the axis label, for example &quot;India (18% of population)&quot;, with an
         # asterisk after the name when the change is clear of measurement error
         country_label = sprintf(&quot;%s%s (%.0f%% of population)&quot;, country,
                                 if_else(clear_of_error, &quot;*&quot;, &quot;&quot;), pop_share_end * 100),
         # order the labels by population, so the largest is at the top of the chart
         country_label = fct_reorder(country_label, pop_share_end))</pre>



<h3 class="wp-block-heading"><em>India was a key driver of global declines in the rule of law</em></h3>



<p class="wp-block-paragraph">Population-weighted average rule of law scores mean more populous countries with larger score changes drive the global average. The code below explores this by looking at the source of declines, with a focus on the five most populated countries (accounting for 46% of world population).</p>



<p class="wp-block-paragraph">The answer is mixed, with India, the United States and Pakistan experiencing declines, Indonesia an improvement and China showing next to no net change. Although India experienced smaller declines relative to the United States and Pakistan, its larger population made it a key driver of declines in the global weighted average rule of law score.</p>



<p class="wp-block-paragraph">When each country’s score is viewed individually, the trends aren’t as clear cut. For instance, although India’s score has declined over the period, the change is within the bounds for measurement error, making it possible that score differences come down to measurement error rather than tangible changes in institutional quality (which is also true for China). On the other hand, the trends exhibited by the United States and Indonesia are large enough to point to real changes over the period: a decline for the former and improvement for the latter.</p>



<p class="wp-block-paragraph"><strong>Figures: Rule of Law in the five most populous countries, 1996 and 2023, and in every year between</strong></p>



<p class="wp-block-paragraph"><strong>Code:</strong> world-trend-contributions</p>



<p class="wp-block-paragraph"><em>The rule of law scores for the five most populous countries: China, India, Indonesia, Pakistan and the United States are then visualized. The first plot shows the change in the WGI RoL score between 1996 and 2023, while the second plot shows how each country’s score has evolved over time, as a smoothed line with uncertainty bands.</em></p>



<pre># For the chart title: the country that moved least, and how many of the other
# four declined. The least-changed country is left out of the count, so the
# title doesn't count it as a fall and also say it barely moved.
dta_populous_least_change &lt;- slice_min(dta_most_populous, abs(wgi_rol_change), n = 1)
sum_populous_nmb_fell &lt;- sum(dta_most_populous$direction == &quot;Decline&quot; &#038;
                             dta_most_populous$iso3c != dta_populous_least_change$iso3c)

plt_populous_change &lt;- dta_most_populous |&gt;
  ggplot(aes(y = country_label)) +
  geom_vline(xintercept = 0, colour = ref_col$grey_pale) +
  # each arrow runs from the country's 1996 score (dot) to its 2023 score;
  # the dot takes the arrow's colour
  geom_point(aes(x = wgi_rol_start, colour = direction), size = 2.5) +
  geom_segment(aes(x = wgi_rol_start, xend = wgi_rol_end, yend = country_label,
                   colour = direction),
               arrow = arrow(length = unit(0.25, &quot;cm&quot;), type = &quot;closed&quot;),
               linewidth = 1.2) +
  # the change is printed beside the arrowhead, on the side it moved towards
  geom_text(aes(x = wgi_rol_end, label = sprintf(&quot;%+.2f&quot;, wgi_rol_change),
                hjust = if_else(direction == &quot;Decline&quot;, 1.4, -0.4)),
            size = 3.2) +
  scale_colour_manual(values = ref_col$direction) +
  scale_x_continuous(expand = expansion(mult = 0.15)) +
  theme(legend.position    = &quot;none&quot;,
        panel.grid.major.y = element_blank(),
        panel.grid.major.x = element_line(colour = ref_col$grey_pale, linewidth = 0.4)) +
  labs(title    = sprintf(&quot;Scores fell in %d of the five most populous countries; %s barely moved&quot;,
                          sum_populous_nmb_fell, dta_populous_least_change$country),
       subtitle = sprintf(&quot;Arrows run from the %d score (dot) to %d; brackets: %d population share&quot;,
                          ref_year$start, ref_year$end, ref_year$end),
       x = paste0(&quot;Rule of Law estimate (WGI units)\n&quot;, ref_lab$axis_score),
       y = NULL, caption = paste(ref_lab$clear_of_error, ref_lab$caption, sep = &quot;\n&quot;))

plt_populous_change</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i2.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-world-trend-contributions-1.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<pre># The same five countries' scores in every year, with each country's colour
# from the chart above, and the largest country in the first panel
dta_populous_yearly &lt;- dta_panel_1996 |&gt;
  filter(iso3c %in% dta_most_populous$iso3c) |&gt;
  left_join(select(dta_most_populous, iso3c, direction, pop_share_end),
            by = join_by(iso3c), relationship = &quot;many-to-one&quot;) |&gt;
  mutate(country = fct_reorder(country, pop_share_end, .desc = TRUE))

# China's lowest score, for the chart title
dta_china_lowest &lt;- dta_populous_yearly |&gt;
  filter(iso3c == &quot;CHN&quot;) |&gt;
  slice_min(wgi_rol, n = 1)

plt_populous_yearly &lt;- dta_populous_yearly |&gt;
  ggplot(aes(x = year, y = wgi_rol, colour = direction, fill = direction)) +
  geom_hline(yintercept = 0, colour = ref_col$grey_pale) +
  # 90% measurement band from the WGI's own standard error for each country and year
  geom_ribbon(aes(ymin = wgi_rol - 1.645 * wgi_rol_se,
                  ymax = wgi_rol + 1.645 * wgi_rol_se),
              colour = NA, alpha = 0.15) +
  # a smoothed line through the yearly scores (loess, ggplot's default for this many points)
  geom_smooth(method = &quot;loess&quot;, formula = y ~ x, se = FALSE, linewidth = 0.9) +
  # one panel per country, all on the same vertical scale
  facet_wrap(~ country, nrow = 1) +
  scale_colour_manual(values = ref_col$direction) +
  scale_fill_manual(values = ref_col$direction) +
  scale_x_continuous(breaks = c(2000, 2010, 2020)) +
  theme(legend.position = &quot;none&quot;) +
  labs(title    = sprintf(&quot;China's small net change hides a fall to %.2f in %d and a recovery since&quot;,
                          dta_china_lowest$wgi_rol, dta_china_lowest$year),
       subtitle = sprintf(&quot;Smoothed WGI RoL score, %d to %d, with 90%% measurement uncertainty bands&quot;,
                          ref_year$start, ref_year$end),
       x = NULL, y = paste0(&quot;Rule of Law estimate (WGI units)\n&quot;, ref_lab$axis_score_vertical),
       caption = ref_lab$caption)

plt_populous_yearly</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i1.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-world-trend-contributions-2.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<h3 class="wp-block-heading"><em>Rule of law scores fell across the majority of region and income groups</em></h3>



<p class="wp-block-paragraph">To get a sense of just how <em>global</em> these <em>global</em> averages are, the next figure presents average population-weighted score changes by region and income classification from 1996 to 2023. Each panel is a region and each arrow is an income group within it. The thin line over each arrow is the 90% measurement error band. Error bands crossing zero point to the direction of change in the rule of law being ambiguous.</p>



<p class="wp-block-paragraph">Overall, countries across almost all regional and income groups experienced declines in their rule of law scores. The exception to this was in East Asia and the Pacific, where three income groups improved, though only one by more than the measurement error bands: the upper middle income group of South Korea, Macao and Palau. The general declines also line up with analysis by the World Justice Project, which found that countries with declines in their rule of law scores outnumbered those with improvements for nine years in a row.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn4" class="footnote-ref" id="fnref4" role="doc-noteref" rel="nofollow" target="_blank"><sup>4</sup></a></p>



<p class="wp-block-paragraph"><strong>Note:</strong> A population-weighted decline doesn’t mean most <em>countries</em> got worse as more populous countries might be driving it (such as India in the chart above).</p>



<p class="wp-block-paragraph"><strong>Figure: Change in population-weighted Rule of Law by starting income group within each region, 1996 to 2023</strong></p>



<p class="wp-block-paragraph"><strong>Code:</strong> plot-arrows-region-income</p>



<p class="wp-block-paragraph"><em>The code below calculates the population-weighted average for each income group within each region, in 1996 and 2023, and plots the change with a 90% measurement band. It also counts the falls that are clear of their band.</em></p>



<pre># Step 1: the population-weighted average for each group in each year, with its
# standard error, worked out as for the world line above
sum_region_income &lt;- dta_panel_1996 |&gt;
  summarise(wgi_rol_popw    = weighted.mean(wgi_rol, w = pop_tot_nmb),
            wgi_rol_popw_se = sqrt(sum((pop_tot_nmb / sum(pop_tot_nmb))^2 * wgi_rol_se^2)),
            nmb_countries   = n(),
            .by = c(region, econ_income_start, year))

# Step 2: copy each group's 1996 average and standard error onto its other
# years, then keep 2023 only, leaving one line per group
sum_region_income &lt;- sum_region_income |&gt;
  mutate(wgi_rol_popw_start    = wgi_rol_popw[year == ref_year$start],
         wgi_rol_popw_se_start = wgi_rol_popw_se[year == ref_year$start],
         .by = c(region, econ_income_start)) |&gt;
  filter(year == ref_year$end)

# Step 3: the change since 1996 and its 90% band. The standard error of the
# change combines the 1996 and 2023 errors, treating them as independent.
sum_region_income &lt;- sum_region_income |&gt;
  mutate(wgi_rol_change    = wgi_rol_popw - wgi_rol_popw_start,
         wgi_rol_change_se = sqrt(wgi_rol_popw_se^2 + wgi_rol_popw_se_start^2),
         change_low        = wgi_rol_change - 1.645 * wgi_rol_change_se,
         change_high       = wgi_rol_change + 1.645 * wgi_rol_change_se)

# Step 4: colour and labels for the chart
sum_region_income &lt;- sum_region_income |&gt;
  mutate(direction = if_else(wgi_rol_change &lt; 0, &quot;Decline&quot;, &quot;Improvement&quot;),
         # the short income group name (e.g. &quot;Lower middle&quot;), keeping the poorest-first order
         econ_income_short = fct_relabel(econ_income_start, \(x) unname(lkp_income$short[x])))

# The axis label: the short income group name, with an asterisk when the
# whole 90% band lies on one side of zero. The labels are ordered poorest
# first, as the income groups are.
sum_region_income &lt;- sum_region_income |&gt;
  mutate(clear_of_error    = change_high &lt; 0 | change_low &gt; 0,
         econ_income_label = paste0(econ_income_short, if_else(clear_of_error, &quot;*&quot;, &quot;&quot;)),
         econ_income_label = fct_reorder(econ_income_label, as.integer(econ_income_short)))

# Falls whose whole 90% band lies below zero, counted in the chart title
sum_region_income_clear &lt;- sum_region_income |&gt;
  filter(change_high &lt; 0)

plt_region_income &lt;- sum_region_income |&gt;
  ggplot(aes(y = econ_income_label, colour = direction)) +
  geom_vline(xintercept = 0, colour = ref_col$grey) +
  # every arrow starts at zero, so only the direction and size of the change show
  geom_segment(aes(x = 0, xend = wgi_rol_change, yend = econ_income_label),
               arrow = arrow(length = unit(0.2, &quot;cm&quot;), type = &quot;closed&quot;),
               linewidth = 1) +
  # 90% measurement band around each change, drawn on top of the arrow in dark
  # grey so both end ticks stay visible, including the one nearest zero
  geom_errorbar(aes(xmin = change_low, xmax = change_high),
                width = 0.5, colour = ref_col$charcoal, linewidth = 0.4) +
  # The label sits beyond the end of the band, on the side the arrow points.
  # The bracketed count shows how many countries sit behind each arrow.
  geom_text(aes(x = if_else(direction == &quot;Decline&quot;, change_low, change_high),
                label = sprintf(&quot;%+.2f (%d)&quot;, wgi_rol_change, nmb_countries),
                hjust = if_else(direction == &quot;Decline&quot;, 1.1, -0.1)),
            size = 3.6) +
  # free_y drops income groups a region doesn't have
  facet_wrap(~ region, ncol = 1, scales = &quot;free_y&quot;) +
  scale_colour_manual(values = ref_col$direction) +
  scale_x_continuous(expand = expansion(mult = 0.2)) +
  # a shaded strip above each panel marks where one region ends and the next begins
  theme(legend.position    = &quot;none&quot;,
        axis.text.y        = element_text(size = rel(1.05)),
        strip.text         = element_text(size = rel(1.05), margin = margin(4, 6, 4, 6)),
        strip.background   = element_rect(fill = ref_col$grey_pale, colour = NA),
        panel.spacing.y    = unit(0.8, &quot;lines&quot;),
        panel.grid.major.y = element_blank(),
        panel.grid.major.x = element_line(colour = ref_col$grey_pale, linewidth = 0.4)) +
  labs(title    = sprintf(&quot;Rule of Law fell in %d of %d groups; only %d falls are clear of measurement error&quot;,
                          sum(sum_region_income$direction == &quot;Decline&quot;),
                          nrow(sum_region_income),
                          nrow(sum_region_income_clear)),
       subtitle = &quot;Arrows: income groups within each region, with 90% bands; countries in brackets&quot;,
       x = paste0(&quot;Change in population-weighted Rule of Law, 1996 to 2023 (WGI units)\n&quot;,
                  ref_lab$axis_change),
       y = &quot;Earliest income group&quot;,
       caption = paste(ref_lab$income_start, ref_lab$clear_of_error, ref_lab$caption, sep = &quot;\n&quot;))

plt_region_income</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i1.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-plot-arrows-region-income-1.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<h2 class="wp-block-heading"><em>Countries with stronger rule of law are richer</em></h2>



<p class="wp-block-paragraph">You don’t need to be an economist to guess that the rule of law is probably stronger in richer countries<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn5" class="footnote-ref" id="fnref5" role="doc-noteref" rel="nofollow" target="_blank"><sup>5</sup></a>, but scatter plots demonstrating this appear in almost every governance report talking about the rule of law. The plot below demonstrates this by comparing each country’s rule of law score in 2023 with its GDP per capita. Purchasing power parity (PPP) terms are used to adjust for differences in purchasing power, such that one PPP dollar has the same <em>real</em> value across countries.</p>



<p class="wp-block-paragraph">The line on the plot shows the <em>typical</em> relationship between the rule of law and average income. Countries to the left of the line are poorer than might be expected given their rule of law scores, while those to the right of the line are wealthier. Although why countries sit on either side is a rabbit hole worth exploring, a large share of the outliers can be explained by geography and the structure of a country’s economy, in particular:</p>



<ul class="wp-block-list">
<li class=""><strong>Countries that are wealthier than might be expected given their rule of law score</strong>: include economies that rely heavily on oil and gas rents, which is theorised to reduce the incentive for government to create inclusive institutions for the benefit of the wider public.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn6" class="footnote-ref" id="fnref6" role="doc-noteref" rel="nofollow" target="_blank"><sup>6</sup></a></li>



<li class=""><strong>Countries that are poorer than might be expected given their rule of law score:</strong> include a large share of Pacific island economies and countries in Sub-Saharan Africa. While the reasons are beyond the scope of this post, Pacific island economies are unusually small and remote, depressing economic activity and trade, while many of the countries in Sub-Saharan Africa face wider institutional and geographic challenges, such as poor governance and having a smaller share of their population near navigable rivers.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn7" class="footnote-ref" id="fnref7" role="doc-noteref" rel="nofollow" target="_blank"><sup>7</sup></a></li>
</ul>



<p class="wp-block-paragraph">As always, my point here isn’t to imply the two things are causally connected as the claim requires more evidence than a pairwise correlation. But, the idea that more secure, predictable and legible property rights <em>might</em> make trade and commerce easier isn’t new, which means there’s plenty of evidence exploring the idea.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn8" class="footnote-ref" id="fnref8" role="doc-noteref" rel="nofollow" target="_blank"><sup>8</sup></a> It’s also a compelling idea to anybody who has spent their career being inundated with governance reports presenting the correlation.</p>



<p class="wp-block-paragraph">But, there’s also an arguable <em>conceptual</em> endogeneity<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn9" class="footnote-ref" id="fnref9" role="doc-noteref" rel="nofollow" target="_blank"><sup>9</sup></a> between GDP per capita and the rule of law. The rule of law plays a central role in deciding what can be owned, priced and traded, while also being shaped by the same political and economic forces it is meant to govern.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn10" class="footnote-ref" id="fnref10" role="doc-noteref" rel="nofollow" target="_blank"><sup>10</sup></a> So by <em>strengthening</em> the rule of law, what can be owned, priced and traded changes too,<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn11" class="footnote-ref" id="fnref11" role="doc-noteref" rel="nofollow" target="_blank"><sup>11</sup></a> thereby expanding what can be counted in GDP statistics. That doesn’t make the connection meaningless as GDP is connected with other things we care about like life expectancy and wellbeing. But it’s good to keep in mind as it’s likely that some part of the correlation reflects their overlapping and mutually reinforcing scope, rather than simply being a clean relationship between two independent variables.</p>



<p class="wp-block-paragraph"><strong>Note:</strong> Oil and gas rents are the value of oil and gas production less what it costs to extract, as a share of GDP, from the World Development Indicators.</p>



<p class="wp-block-paragraph"><strong>Code:</strong> levels-data</p>



<p class="wp-block-paragraph"><em>The code below produces a scatter plot comparing a country’s rule of law score with GDP per capita in 2023. The size of each country’s point is scaled by population, while economies with significant oil rents (>10%), Pacific island small states and Sub-Saharan African countries are coloured.</em></p>



<pre># Every country with a score and an income in 2023
dta_rol_income &lt;- dta_panel |&gt;
  filter(year == ref_year$end, !is.na(wgi_rol), !is.na(econ_gdppc_ppp))

# Correlation between the score and log income, for the chart title
sum_cor_rol_income &lt;- cor(dta_rol_income$wgi_rol, log(dta_rol_income$econ_gdppc_ppp))

# Oil and gas rents for each country, averaged over 2019 to 2023 because they
# swing with prices. Countries with no rents data get NaN (not a number).
dta_oilgas_rents &lt;- dta_panel |&gt;
  filter(year %in% ref_year$rents_window) |&gt;
  summarise(res_oilgas_pct = mean(res_oil_rents_pct + res_gas_rents_pct, na.rm = TRUE),
            .by = iso3c)

# The World Bank's Pacific island small states (group PSS), taken from the
# &quot;composition&quot; sheet of its CLASS workbook, downloaded 7 November 2026
ref_pacific_islands &lt;- read_csv(ref_path$pacific, show_col_types = FALSE)

dta_rol_income &lt;- dta_rol_income |&gt;
  left_join(dta_oilgas_rents, by = join_by(iso3c), relationship = &quot;one-to-one&quot;)

# The colour group of each country on the chart. case_when() takes the first rule
# that is true, so an African producer counts as a producer. Countries with no
# rents data are not producers.
dta_rol_income &lt;- dta_rol_income |&gt;
  mutate(point_group = case_when(
    res_oilgas_pct &gt; ref_cutoff$large_producer_rents_pct ~ ref_lab$large_producers,
    iso3c %in% ref_pacific_islands$iso3c                 ~ &quot;Pacific island small states&quot;,
    region == &quot;Sub-Saharan Africa&quot;                       ~ &quot;Sub-Saharan Africa&quot;,
    .default = &quot;Other countries&quot;))

# The colour of each group on the chart; this order is also the legend order
lkp_point_colours &lt;- c(ref_col$orange, ref_col$navy, ref_col$blue, ref_col$grey_light)
names(lkp_point_colours) &lt;- c(ref_lab$large_producers, &quot;Pacific island small states&quot;,
                              &quot;Sub-Saharan Africa&quot;, &quot;Other countries&quot;)

plt_rol_income &lt;- dta_rol_income |&gt;
  ggplot(aes(x = econ_gdppc_ppp, y = wgi_rol)) +
  # a straight line of log income on the score; orientation = &quot;y&quot; fits income on the
  # score, so a country to the right of the line is richer than its score suggests
  geom_smooth(method = &quot;lm&quot;, formula = y ~ x, orientation = &quot;y&quot;, se = FALSE,
              colour = ref_col$grey, linewidth = 0.7) +
  geom_point(aes(size = pop_tot_nmb, colour = point_group), alpha = 0.6) +
  scale_x_log10(labels = scales::label_dollar()) +
  scale_colour_manual(values = lkp_point_colours, breaks = names(lkp_point_colours)) +
  guides(colour = guide_legend(override.aes = list(size = 3, alpha = 1))) +
  # a minimum point size keeps the smallest countries, such as the Pacific islands, visible
  scale_size(range = c(1.5, 12), guide = &quot;none&quot;) +
  labs(title    = sprintf(&quot;Richer countries score higher on the Rule of Law (correlation %.2f)&quot;,
                          sum_cor_rol_income),
       subtitle = paste0(&quot;Countries in &quot;, ref_year$end,
                         &quot;, sized by population; the line is the income typical of each score&quot;),
       x = &quot;GDP per capita (PPP, constant international $, log scale)&quot;,
       y = paste0(&quot;Rule of Law estimate (WGI units)\n&quot;, ref_lab$axis_score_vertical),
       caption = paste(ref_lab$caption, &quot;World Bank country classifications.&quot;))

plt_rol_income</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i2.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-levels-data-1.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<h2 class="wp-block-heading"><em>But economic growth without the rule of law exists</em></h2>



<p class="wp-block-paragraph">Almost every discussion of the links between institutions and economic growth eventually turns to China. China, so the story goes, has grown quickly without investing in pesky institutions like the <em>rule of law</em>.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn12" class="footnote-ref" id="fnref12" role="doc-noteref" rel="nofollow" target="_blank"><sup>12</sup></a> This section tests that story in two steps: first by comparing growth with where countries started on the rule of law in 1996, and then by comparing growth with how their rule of law changed over the period.</p>



<p class="wp-block-paragraph">The plot below shows the distribution of average GDP per capita growth from 1997 to 2023, with countries grouped into four groups based on their rule of law score in 1996. It’s as good a time as any to repeat the warning about drawing sweeping conclusions from observational analysis like this, but the data suggests two things about economic growth and the rule of law:</p>



<ol class="wp-block-list">
<li class=""><strong>Strong growth is possible where the rule of law is weak:</strong> many of the countries that started with weak scores grew faster than those that started with strong ones; and</li>



<li class=""><strong>A strong rule of law doesn’t guarantee high growth:</strong> the countries with the strongest rule of law scores in 1996 had the lowest median growth of the four groups.<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn13" class="footnote-ref" id="fnref13" role="doc-noteref" rel="nofollow" target="_blank"><sup>13</sup></a></li>
</ol>



<p class="wp-block-paragraph"><strong>Note:</strong> Most of the countries in the strongest quarter were already high income at the start of the period, and richer countries tend to grow more slowly, partly because they have less room to catch up.</p>



<p class="wp-block-paragraph"><strong>Figure: Average growth 1997–2023 by Rule of Law in 1996</strong></p>



<p class="wp-block-paragraph"><strong>Code:</strong> plot-growth-vs-start</p>



<p class="wp-block-paragraph"><em>The code below calculates the average annual economic growth and presents this as a histogram with countries grouped according to their rule of law score in 1996.</em></p>



<pre># One line per country: its score in 1996 and 2023, and its average growth.
# Growth in 1996 itself happened before the starting score, so it is left out.
# first() returns the single value for that year, or NA if the country has none.
# Some countries miss a few years of growth, hence na.rm = TRUE.
dta_country_growth &lt;- dta_panel |&gt;
  summarise(
    wgi_rol_start       = first(wgi_rol[year == ref_year$start]),
    wgi_rol_end         = first(wgi_rol[year == ref_year$end]),
    wgi_rol_se_start    = first(wgi_rol_se[year == ref_year$start]),
    wgi_rol_se_end      = first(wgi_rol_se[year == ref_year$end]),
    econ_growth_avg_pct = mean(econ_gdppc_growth_pct[year &gt; ref_year$start], na.rm = TRUE),
    nmb_growth_years    = sum(!is.na(econ_gdppc_growth_pct[year &gt; ref_year$start])),
    .by = c(iso3c, country, region))

# Keep countries with a 1996 score and at least 20 years of growth data, then
# add the change in score and the starting income group
dta_country_growth &lt;- dta_country_growth |&gt;
  filter(!is.na(wgi_rol_start),
         nmb_growth_years &gt;= ref_cutoff$min_growth_years) |&gt;
  mutate(wgi_rol_change = wgi_rol_end - wgi_rol_start) |&gt;
  left_join(lkp_income_start, by = join_by(iso3c), relationship = &quot;one-to-one&quot;)

# Panel titles, weakest starting score first
lkp_quarter_titles &lt;- c(&quot;Weakest quarter of 1996 Rule of Law scores&quot;, &quot;Second-weakest quarter&quot;,
                 &quot;Second-strongest quarter&quot;, &quot;Strongest quarter of 1996 Rule of Law scores&quot;)

# ntile() splits the countries into four equal-sized groups by their 1996 score,
# 1 being the weakest. rev() reverses the panel order only, so the strongest
# quarter is drawn at the top.
dta_country_growth &lt;- dta_country_growth |&gt;
  mutate(start_quarter       = ntile(wgi_rol_start, 4),
         start_quarter_label = factor(lkp_quarter_titles[start_quarter], levels = rev(lkp_quarter_titles)))

# Median, spread and counts for each quarter, used in the chart and the note
# above it (how many in each quarter started as high income)
sum_growth_quarter &lt;- dta_country_growth |&gt;
  summarise(nmb_countries   = n(),
            nmb_high_income = sum(econ_hic_start == &quot;High income&quot;),
            growth_median   = median(econ_growth_avg_pct),
            growth_sd       = sd(econ_growth_avg_pct),
            growth_p10      = quantile(econ_growth_avg_pct, 0.1),
            growth_p90      = quantile(econ_growth_avg_pct, 0.9),
            .by = c(start_quarter, start_quarter_label)) |&gt;
  arrange(start_quarter)

sum_growth_quarter
# A tibble: 4 × 8
  start_quarter start_quarter_label  nmb_countries nmb_high_income growth_median
          &lt;int&gt; &lt;fct&gt;                        &lt;int&gt;           &lt;int&gt;         &lt;dbl&gt;
1             1 Weakest quarter of …            48               0          1.89
2             2 Second-weakest quar…            48               0          2.31
3             3 Second-strongest qu…            48               5          2.01
4             4 Strongest quarter o…            47              31          1.29
# &#x2139; 3 more variables: growth_sd &lt;dbl&gt;, growth_p10 &lt;dbl&gt;, growth_p90 &lt;dbl&gt;
plt_growth_quarter &lt;- dta_country_growth |&gt;
  ggplot(aes(x = econ_growth_avg_pct)) +
  # shaded band: the middle 80% of countries in each quarter
  geom_rect(data = sum_growth_quarter, inherit.aes = FALSE,
            aes(xmin = growth_p10, xmax = growth_p90, ymin = -Inf, ymax = Inf),
            fill = ref_col$blue, alpha = 0.1) +
  geom_histogram(binwidth = 0.5, boundary = 0, fill = ref_col$blue, colour = &quot;white&quot;) +
  geom_vline(xintercept = 0, colour = ref_col$grey) +
  # the median of each quarter
  geom_vline(data = sum_growth_quarter, aes(xintercept = growth_median),
             colour = ref_col$navy, linewidth = 0.9) +
  facet_wrap(~ start_quarter_label, ncol = 1) +
  theme(panel.grid.major.x = element_line(colour = ref_col$grey_pale, linewidth = 0.4)) +
  labs(title    = sprintf(&quot;Growth varied most among weak starters (SD %.1f points) and least among strong (%.1f)&quot;,
                          sum_growth_quarter$growth_sd[1], sum_growth_quarter$growth_sd[4]),
       subtitle = &quot;Shading spans the middle 80% of countries; the dark line is the median&quot;,
       x = &quot;Average GDP per capita growth, 1997–2023 (% a year)&quot;,
       y = &quot;Countries&quot;, caption = ref_lab$caption)

plt_growth_quarter</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i2.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-plot-growth-vs-start-1.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<h3 class="wp-block-heading"><em>Countries where the rule of law improved were more likely to grow faster</em></h3>



<p class="wp-block-paragraph">The histogram above looked at growth against a country’s <em>level</em> of rule of law in 1996. The plot below looks at growth against the <em>change</em> in its rule of law score between 1996 and 2023.</p>



<p class="wp-block-paragraph">In essence, it asks whether countries whose rule of law improved more also grew faster. The answer is a <em>qualified</em> yes, and the relationship is positive in every region with enough countries to measure it.</p>



<p class="wp-block-paragraph">This isn’t to say that improvements in the rule of law <em>cause</em> economic growth<a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fn14" class="footnote-ref" id="fnref14" role="doc-noteref" rel="nofollow" target="_blank"><sup>14</sup></a>, only that the two have moved together over the last three decades. Causality could run the other way, with faster-growing countries able to spend more on the courts, police and other systems that support the rule of law, or it could run both ways. Other drivers of growth may also be missing from the analysis, such as the quality of regulation and political accountability, which <a href="https://www.gilesd-j.com/2026/08/10/institutions-are-a-package-deal-what-a-correlation-cant-tell-you-about-the-rule-of-law/" rel="nofollow" target="_blank">are closely tied to the rule of law</a>. Even so, the pattern helps explain why economists are so excited to talk about how and why the two might be linked.</p>



<p class="wp-block-paragraph">The plot also highlights countries that grew faster than might be expected given changes in their score. China, the example that everyone raises, is there but isn’t alone: Equatorial Guinea, Guyana, Myanmar, Armenia, and Bosnia and Herzegovina all sit well above the line with no clear change in their rule of law score. Several have likely explanations, such as oil discoveries, economic recovery after war and the breakdown of the Soviet Union. China’s small change over the period also hides a fall in its score in the mid-2000s which was followed by a long recovery. None of this is to deny that China holds lessons about institutions and growth (it does), only that it’s not the only outlier that’s worth explaining.</p>



<p class="wp-block-paragraph"><strong>Note:</strong> North America showed the opposite relationship but is left off the regional chart, as it has only three countries (Bermuda, Canada and the United States).</p>



<p class="wp-block-paragraph"><strong>Figures: Change in Rule of Law and average growth, 1996–2023, for all countries and by region</strong></p>



<p class="wp-block-paragraph"><strong>Code:</strong> plot-growth-vs-change</p>



<p class="wp-block-paragraph"><em>The code below produces two plots: a scatter plot comparing economic growth with changes in the rule of law, with the unusually fast growers whose score didn’t clearly change named, and a bar plot that reports the correlation between the two variables within each region.</em></p>



<pre># The countries in the histograms that have a score in 2023 as well as 1996
dta_growth_change &lt;- dta_country_growth |&gt;
  filter(!is.na(wgi_rol_change))

# How far each country's growth sits above or below the straight line drawn
# through all countries, in percentage points
dta_growth_change &lt;- dta_growth_change |&gt;
  mutate(growth_vs_line = residuals(lm(econ_growth_avg_pct ~ wgi_rol_change)))

# The countries named on the chart: growth more than two standard deviations
# of these distances above the line, and a change in score inside its 90%
# measurement band, worked out as for the five most populous countries
dta_growth_change &lt;- dta_growth_change |&gt;
  mutate(wgi_rol_change_se  = sqrt(wgi_rol_se_start^2 + wgi_rol_se_end^2),
         change_within_band = abs(wgi_rol_change) &lt;= 1.645 * wgi_rol_change_se,
         fast_no_change     = growth_vs_line &gt; ref_cutoff$outlier_sd * sd(growth_vs_line) &
                              change_within_band)

plt_growth_change &lt;- dta_growth_change |&gt;
  ggplot(aes(x = wgi_rol_change, y = econ_growth_avg_pct)) +
  geom_hline(yintercept = 0, colour = ref_col$grey_pale) +
  geom_vline(xintercept = 0, colour = ref_col$grey_pale) +
  geom_point(colour = ref_col$blue, alpha = 0.6) +
  # one straight-line fit through all countries, with no controls
  geom_smooth(method = &quot;lm&quot;, formula = y ~ x, se = FALSE,
              colour = ref_col$grey, linewidth = 0.8) +
  # name the fast growers whose score didn't clearly change, China among them.
  # Each label is joined to its point by a short line (min.segment.length = 0 always draws it).
  geom_text_repel(data = filter(dta_growth_change, fast_no_change),
                  aes(label = country), size = 3, colour = ref_col$charcoal,
                  family = &quot;Open Sans&quot;, seed = 1, min.segment.length = 0) +
  labs(title    = &quot;Countries whose Rule of Law improved grew faster than those where it declined&quot;,
       subtitle = &quot;Each country's average growth against its change in score; straight-line fit, no controls&quot;,
       x = paste0(&quot;Change in Rule of Law, 1996–2023 (WGI units)\n&quot;, ref_lab$axis_change),
       y = &quot;Average GDP per capita growth, 1997–2023 (% a year)&quot;,
       caption = paste(sprintf(&quot;Named: growth more than %d standard deviations above the line, and a change in score inside its 90%% measurement band.&quot;,
                               ref_cutoff$outlier_sd),
                       ref_lab$caption, sep = &quot;\n&quot;))

plt_growth_change</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i0.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-plot-growth-vs-change-1.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<pre># Correlation between the change in score and average growth within each region
sum_cor_region &lt;- dta_growth_change |&gt;
  summarise(nmb_countries = n(),
            cor_change    = cor(wgi_rol_change, econ_growth_avg_pct),
            .by = region)

# Regions with fewer countries than the cut-off are left off the chart. On the
# data used here (checked 10 November 2026) this is North America (3 countries:
# Bermuda, Canada and the United States).
sum_cor_small_regions &lt;- sum_cor_region |&gt;
  filter(nmb_countries &lt; ref_cutoff$min_group_countries)

sum_cor_small_regions
# A tibble: 1 × 3
  region        nmb_countries cor_change
  &lt;chr&gt;                 &lt;int&gt;      &lt;dbl&gt;
1 North America             3     -0.943
# Keep the regions large enough to chart
sum_cor_region &lt;- sum_cor_region |&gt;
  filter(nmb_countries &gt;= ref_cutoff$min_group_countries)

# Is each correlation statistically significant at the 10% level, matching the
# 90% bands used elsewhere in the post? This is the standard test that the
# correlation is zero, written out: the t statistic is
# r x sqrt((n - 2) / (1 - r^2)), with n - 2 degrees of freedom. It gives the
# same p-value as cor.test().
sum_cor_region &lt;- sum_cor_region |&gt;
  mutate(cor_t       = cor_change * sqrt((nmb_countries - 2) / (1 - cor_change^2)),
         cor_p_value = 2 * pt(-abs(cor_t), df = nmb_countries - 2),
         cor_signif  = cor_p_value &lt; 0.10)

# The labels: an asterisk after the region when the correlation is significant,
# and the number of countries in brackets
sum_cor_region &lt;- sum_cor_region |&gt;
  mutate(region_label = sprintf(&quot;%s%s (%d)&quot;, region, if_else(cor_signif, &quot;*&quot;, &quot;&quot;), nmb_countries),
         # largest correlation at the top
         region_label = fct_reorder(region_label, cor_change))

plt_cor_region &lt;- sum_cor_region |&gt;
  ggplot(aes(x = cor_change, y = region_label)) +
  geom_col(fill = ref_col$blue, width = 0.6) +
  geom_vline(xintercept = 0, colour = ref_col$grey) +
  # the value sits just beyond the end of each bar
  geom_text(aes(label = sprintf(&quot;%.2f&quot;, cor_change)), hjust = -0.2, size = 3.2) +
  scale_x_continuous(expand = expansion(mult = c(0, 0.1))) +
  theme(panel.grid.major.y = element_blank(),
        panel.grid.major.x = element_line(colour = ref_col$grey_pale, linewidth = 0.4)) +
  labs(title    = sprintf(&quot;Improvement went with faster growth in %d of %d regions&quot;,
                          sum(sum_cor_region$cor_change &gt; 0),
                          nrow(sum_cor_region)),
       subtitle = sprintf(&quot;Correlation within each region, countries in brackets; regions under %d countries left out&quot;,
                          ref_cutoff$min_group_countries),
       x = &quot;Correlation between change in Rule of Law and average growth, 1996–2023&quot;,
       y = NULL, caption = paste(&quot;* Statistically significant at the 10% level: the correlation's 90% interval excludes zero.&quot;,
                                 ref_lab$caption, sep = &quot;\n&quot;))

plt_cor_region</pre>



<figure class="wp-block-image size-full"><img decoding="async" loading="lazy" src="https://i0.wp.com/www.gilesd-j.com/wp-content/uploads/2026/11/261017-plot-growth-vs-change-2.png?w=578&#038;ssl=1" alt="" data-recalc-dims="1"/></figure>



<h2 class="wp-block-heading">Summing up</h2>



<p class="wp-block-paragraph">The point of this post, besides satisfying my curiosity about the data, was to lay out a set of common themes that emerge in the literature:</p>



<ul class="wp-block-list">
<li class="">Globally, the rule of law has deteriorated since the mid 90s;</li>



<li class="">The rule of law and economic growth appear to be linked;</li>



<li class="">The rule of law’s relationship with economic growth is positive across regions; and</li>



<li class="">The rule of law isn’t a necessary or sufficient condition for economic growth.</li>
</ul>



<p class="wp-block-paragraph">If you’re familiar with the field, none of this will be particularly surprising, but each of these themes provides useful context for future posts where I attempt to tease out some of these relationships and trends in greater detail. But, I also wanted to make a contribution to the grey literature on the topic as from my observations there are few explorations of the topic that dig into the technical details outside published papers on the topic.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<ol class="wp-block-list">
<li class=""><p>World Bank (2025). “The Worldwide Governance Indicators: revised methodology for measuring governance using perception data.” December 2025. Prepared by C. N. Jiang, A. Jain, T. Romano and T. Boyce.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref1" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>Gallup, J. L., Sachs, J. D. and Mellinger, A. D. (1999). “Geography and economic development.” <em>International Regional Science Review</em> 22(2): 179–232.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref2" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>Kaufmann, D., Kraay, A. and Mastruzzi, M. (2010). “The Worldwide Governance Indicators: methodology and analytical issues.” World Bank Policy Research Working Paper 5430.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref3" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>World Justice Project (2026). <em>WJP Rule of Law Index 2026</em>. Washington, DC: World Justice Project. Released 21 October 2026. <a href="https://worldjusticeproject.org/rule-of-law-index/" rel="nofollow" target="_blank">worldjusticeproject.org/rule-of-law-index</a><a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref4" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>Analysis also points to the correlation becoming stronger over time, see: Bhagat, S., 2020. Economic growth, income inequality, and the rule of law. <em>Harvard Business Law Review Online</em>.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref5" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>Natural Resource Governance Institute 2015, <em>The resource curse: The political and economic challenges of natural resource wealth</em> , NRGI Reader, viewed 7 November 2026, <a href="https://resourcegovernance.org/analysis-tools/publications/primer-resource-curse" rel="nofollow" target="_blank">resourcegovernance.org</a>.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref6" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>Distance from the sea and access to inland rivers may also play a part. Countries where fewer than a quarter of people live within 100 km of a coast or navigable river are mostly poorer than their score predicts (many of them are in Sub-Saharan Africa). I tested this using Gallup, Sachs and Mellinger’s geography data (<a href="https://ideas.repec.org/p/boc/bocins/geodata.html" rel="nofollow" target="_blank">RePEc</a>), but left the analysis and grouping off the chart as it largely picks out countries in Sub-Saharan Africa and I didn’t want to imply there was a simple explanation for something this complicated. In my defence, I’m in good company: Adam Smith argued that water transport widens markets and Jeffrey Sachs and co-authors make a similar case for navigable water, see: Mellinger, A.D., Sachs, J.D. and Gallup, J.L., 1999. Climate, water navigability, and economic development. CID Working Paper No. 24, Center for International Development, Harvard University.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref7" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>For instance, see: Rigobon, R. and Rodrik, D., 2005. Rule of law, democracy, openness, and income: Estimating the interrelationships. <em>Economics of transition</em>, <em>13</em>(3), pp.533-564; Haggard, S. and Tiede, L., 2011. The rule of law and economic growth: where are we?. World development, 39(5), pp.673-685; and Butkiewicz, J.L. and Yanikkaya, H., 2006. Institutional quality and economic growth: Maintenance of the rule of law or democratic institutions, or both?. Economic Modelling, 23(4), pp.648-661.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref8" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>In addition to the <em>statistical</em> endogeneity, such as wealthier nations being more able to invest in their legal and regulatory systems.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref9" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>e.g. see: Krever, T., 2011. The legal turn in late development theory: The rule of law and the World Bank’s development model. <em>Harv. Int’l LJ</em>, <em>52</em>, p.287; and Haggard, S., MacIntyre, A. and Tiede, L., 2008. The rule of law and economic development. <em>Annu. Rev. Polit. Sci.</em>, <em>11</em>(1), pp.205-234.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref10" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>Pistor makes a similar point in her book ‘The Code of Capital’, noting that: the law allows the attachment of “…wealth generating attributes, namely priority, durability, convertibility, and universality onto different assets.” See: Pistor, K., 2019. <em>The code of capital: How the law creates wealth and inequality</em>. Princeton University Press.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref11" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>For a good overview of why it’s thought China has achieved this, see: Lopez-Gomez, L. (2026) The Rule of Law and the Institutional Roots of Economic Performance: Literature Review. London: Bingham Centre for the Rule of Law and the Law Society of England and Wales. Available at: <a href="https://binghamcentre.biicl.org/publications/the-rule-of-law-and-the-institutional-roots-of-economic-performance" class="uri" rel="nofollow" target="_blank">https://binghamcentre.biicl.org/publications/the-rule-of-law-and-the-institutional-roots-of-economic-performance</a>, p.17<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref12" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>The non-linear relationship between the rule of law and other institutional variables is well-documented, e.g. Beyaert, A., García-Solanes, J. and Lopez-Gomez, L., 2023. Corruption, quality of institutions and growth. <em>Applied economic analysis</em>, <em>31</em>(91), pp.55-72.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref13" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>



<li class=""><p>This isn’t to say evidence doesn’t exist: for instance, Besley et al. (2025) found that areas in England with greater legal capacity during the industrial revolution achieved better social, economic and institutional outcomes. Because it’s plausible that legal capacity has been exogenously determined, the authors believe the association can be interpreted as being causal. See: Besley, T., Bogart, D., Chapman, J. and Palma, N., 2025. Justices of the peace: Legal foundations of the industrial revolution (No. 20214). Centre for Economic Policy Research.<a role="doc-backlink" href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/#fnref14" class="footnote-back" rel="nofollow" target="_blank"><img src="https://i2.wp.com/s.w.org/images/core/emoji/17.0.2/72x72/21a9.png?w=578&#038;ssl=1" alt="&#x21a9;" class="wp-smiley" style="height: 1em; max-height: 1em;" data-recalc-dims="1" />︎</a></p></li>
</ol>
<p>The post <a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/" rel="nofollow" target="_blank">The rule of law and economic growth: what three decades of data show</a> appeared first on <a href="https://www.gilesd-j.com/" rel="nofollow" target="_blank">Giles</a>.</p>

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.gilesd-j.com/2026/11/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/"> Data Analytics and AI Archives - Giles</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/the-rule-of-law-and-economic-growth-what-three-decades-of-data-show/">The rule of law and economic growth: what three decades of data show</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404779</post-id>	</item>
		<item>
		<title>Agents in Production: Build an AI Chat App with Shiny and {shinychat}</title>
		<link>https://www.r-bloggers.com/2026/11/agents-in-production-build-an-ai-chat-app-with-shiny-and-shinychat/</link>
		
		<dc:creator><![CDATA[The Jumping Rivers Blog]]></dc:creator>
		<pubDate>Tue, 10 Nov 2026 23:59:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://www.jumpingrivers.com/blog/agents-in-production-shiny/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>The agent we built in part 1 lives in an R console, which suits one person. A trial team needs an app in the browser, where several people can ask questions at once without seeing each other’s conversations.<br />
This is part two of a five-part ...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/agents-in-production-build-an-ai-chat-app-with-shiny-and-shinychat/">Agents in Production: Build an AI Chat App with Shiny and {shinychat}</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://www.jumpingrivers.com/blog/agents-in-production-shiny/"> The Jumping Rivers Blog</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>

<p>
<a href = "https://www.jumpingrivers.com/blog/agents-in-production-shiny/">
<img src="https://www.jumpingrivers.com/blog/agents-in-production-shiny/" width="400" style="width:400px" class="image-center" style="display: block; margin: auto;" />
</a>
</p>
<p>The agent we built in <a href="https://www.jumpingrivers.com/blog/agents-in-production-first-agent/" rel="nofollow" target="_blank">part 1</a> lives in an R console, which suits one person. A trial team needs an app in the browser, where several people can ask questions at once without seeing each other’s conversations.</p>
<p>This is part two of a five-part series on putting AI agents into production with R.
Each part builds on the same app, a clinical trial safety assistant. The parts are:</p>
<ul>
<li><a href="https://www.jumpingrivers.com/blog/agents-in-production-first-agent/" rel="nofollow" target="_blank">From Chatbot to Agent</a></li>
<li><a href="https://www.jumpingrivers.com/blog/agents-in-production-shiny/" rel="nofollow" target="_blank">Putting the Agent in a Shiny App</a> (this post)</li>
<li>Can You Trust It?</li>
<li>Audit Trails, Costs and Models</li>
<li>Sharing Your Tools with MCP</li>
</ul>
<p>As a reminder, the agent answers questions about a clinical trial’s safety data with a model, four tools and a loop. The model chooses which tool to call, and the tools do the counting in R. The model is also told never to do the maths itself, and part 3 checks that. (Part 1 also explains <a href="https://www.jumpingrivers.com/blog/agents-in-production-first-agent/#trial-terms-in-plain-english" rel="nofollow" target="_blank">the trial terms</a>.)</p>
<p>In this post we:</p>
<ul>
<li>build a chat app around the agent with {shiny} and {shinychat}</li>
<li>give every user their own conversation, and test it</li>
<li>show the agent’s tool calls in the chat</li>
<li>stop files being uploaded to the model</li>
<li>handle the model being slow or unavailable, and deploy to Posit Connect</li>
</ul>
<aside class="advert">
<p>
Want to take your agent into production? Join Pedro for our free <a href="https://www.jumpingrivers.com/webinars/?utm_source=blog&#038;utm_medium=banner&#038;utm_campaign=2026-agents-in-production-shiny" rel="nofollow" target="_blank">Agents in Production webinar</a> on Thursday 19 November 2026 at 1:15 PM UK time, covering testing, logging and safeguards.
</p>
</aside>
<h2 id="what-is-shinychat">What is {shinychat}?</h2>
<p><a href="https://posit-dev.github.io/shinychat/" rel="nofollow" target="_blank">{shinychat}</a> is Posit’s package of chat components for Shiny, and we need two of them. The chat window comes from <code>chat_ui()</code>. Behind it, <code>chat_server()</code> connects the window to an {ellmer} chat client. It sends each question to the model, streams the answer back and shows each tool call as it happens.</p>
<p>The client is the one part 1’s <code>make_agent()</code> builds, so the tools, the system prompt and the loop all carry over unchanged. If you’ve used <code>chat_mod_ui()</code> and <code>chat_mod_server()</code> before, {shinychat} 0.5.0 soft-deprecates them in favour of these two.</p>
<h2 id="the-app">The app</h2>
<p>We keep the page in its own function, <code>app_ui()</code>, so that tests can check it without starting the app:</p>
<!-- source: code/R/ui.R, GREETING and app_ui() -->
<pre>GREETING &lt;- paste(
 &quot;Ask me about adverse events in CDISC pilot study 01: a xanomeline skin patch at two doses, against placebo.&quot;,
 &quot;For example: *Which adverse events were most common on the high dose?*&quot;,
 &quot;I only know this trial&#39;s safety data, and I can&#39;t give medical advice.&quot;,
 sep = &quot;\n\n&quot;
)

app_ui &lt;- function() {
 bslib::page_fillable(
 title = &quot;Trial safety assistant&quot;,
 shinychat::chat_ui(
 &quot;chat&quot;,
 greeting = GREETING,
 placeholder = &quot;Ask about the trial&#39;s adverse events&quot;,
 allow_attachments = FALSE
 )
 )
}
</pre><p>The greeting tells people what the assistant is for. It only sets expectations, though. The rule against medical advice goes into the system prompt in part 3, where we also test that the agent keeps it.</p>
<p>The app itself is short, because the work happens in the files in <code>R/</code>: part 1’s, plus four new ones (<code>ui.R</code>, <code>display.R</code>, <code>wrap.R</code> and <code>attachments.R</code>), which Shiny sources before <code>app.R</code> runs:</p>
<!-- source: code/app.R -->
<pre>library(shiny)
library(bslib)
library(shinychat)

# shinychat writes a failed answer&#39;s error into the chat. Sanitised, it says
# &quot;An error occurred&quot; instead of showing the provider&#39;s raw message.
options(shiny.sanitize.errors = TRUE)

# Shiny sources every file in R/ before this runs.
trial_data &lt;- load_trial_data()

ui &lt;- app_ui()

server &lt;- function(input, output, session) {
 # One client per session. A client made outside server() would share one
 # conversation between every user.
 client &lt;- make_agent(tools = with_display(trial_tools(trial_data)))
 refuse_attachments(client)
 chat &lt;- chat_server(&quot;chat&quot;, client, history = FALSE)

 observeEvent(chat$last_error(), {
 error &lt;- chat$last_error()
 message &lt;- if (inherits(error, &quot;trial_attachment_refused&quot;)) {
 conditionMessage(error)
 } else {
 &quot;The assistant couldn&#39;t answer just now. Please try again in a minute.&quot;
 }
 showNotification(message, type = &quot;error&quot;, duration = NULL)
 })
}

shinyApp(ui, server)
</pre><p>For each person who opens the app, <code>server()</code> does four things:</p>
<ul>
<li><strong>It makes them their own chat client.</strong> <code>make_agent()</code> builds it with the four tools, each wrapped by <code>with_display()</code> so that its results read well in the chat.</li>
<li><strong>It checks every request before it goes to the model</strong>, with <code>refuse_attachments()</code>.</li>
<li><strong>It connects the client to the chat window.</strong> That’s <code>chat_server()</code>, with its built-in conversation history turned off (<code>history = FALSE</code>). Left on, {shinychat} 0.5.0 saves every conversation as files on the server, patient records included, outside the audit log and any retention rule, and asks the model to title each one. Part 4 keeps one record instead.</li>
<li><strong>It tells the user what to do if a question fails</strong>, by watching <code>chat$last_error()</code>.</li>
</ul>
<p>The data is loaded once, outside <code>server()</code>, and shared by every session. That’s safe because nothing changes it.</p>
<p>To run the app, get this part’s folder as in part 1 and open it as a project. Your key doesn’t come with it, since git ignores <code>.Renviron</code>, so copy that file across from part 1’s folder.</p>
<p>Then run <code>renv::restore()</code>, which is quick because {renv} reuses the packages it installed before, or install the new ones with <code>install.packages(c(&quot;shiny&quot;, &quot;shinychat&quot;, &quot;rsconnect&quot;))</code>. Finally, call <code>shiny::runApp()</code>.</p>
<!-- Screenshot taken 2026-09-28 with gemini-3.6-flash, as gemini-3.7-flash was out of free quota. It is 740 px wide: when it is retaken on gemini-3.7-flash, use device scale factor 2 (about 1480 px wide). -->
<figure class="w600">
<figcaption>The app answering a question. The "Summarising adverse events" row above the answer is the tool call; open it to see what the agent asked for and what came back.</figcaption>
<img src="https://www.jumpingrivers.com/blog/agents-in-production-shiny/assets/app-answer.webp" alt="Screenshot of the trial safety assistant. The user asks which adverse events were most common on the high dose. Above the answer, a collapsed row reads 'Summarising adverse events'. The answer lists the ten most common events in the Xanomeline High Dose arm of 72 subjects, from pruritus in 25 subjects (34.7%) to application site dermatitis in 7 subjects (9.7%).">
</figure>
<h2 id="one-conversation-per-user">One conversation per user</h2>
<p>An {ellmer} chat client sends the conversation so far with each new question. That’s what makes follow-ups work, and it means a client belongs to one person.</p>
<p>Shiny runs one R process for many users, and it calls <code>server()</code> once for each browser session. Anything created inside <code>server()</code> belongs to that session, while anything created at the top of <code>app.R</code> is shared by everyone.</p>
<p>So a client created at the top of the file puts every user in the same conversation. The second person to arrive can ask “what did I just ask?” and get the first person’s question back.</p>
<figure class="w600">
<figcaption>Each browser session gets its own chat client. Only the trial data, which never changes, is shared.</figcaption>
<a href="https://www.jumpingrivers.com/blog/agents-in-production-shiny/assets/one-client-per-session.svg" rel="nofollow" target="_blank"><img src="https://www.jumpingrivers.com/blog/agents-in-production-shiny/assets/one-client-per-session.svg" alt="Diagram. Two browsers connect to one Shiny app running in one R process. Browser 1 connects to session 1 and browser 2 to session 2; each session has its own chat client and its own conversation. Both clients call the same model through its API. The trial data sits in the app, loaded once and read-only. A note says a client created outside server() would be shared by every session, so browser 2 would talk into browser 1's conversation."></a>
</figure>
<p>You won’t catch this by trying the app on your own, so we test for it. Shiny’s <code>testServer()</code> runs the server function without a browser. Our test builds one app with <code>shiny::shinyAppDir()</code>, runs <code>testServer()</code> on it twice, and checks that the two sessions never share a <code>chat$client</code>.</p>
<p>For the browser tests that go alongside <code>testServer()</code>, see our <a href="https://www.jumpingrivers.com/blog/end-to-end-testing-shinytest2-part-1/" rel="nofollow" target="_blank">series on {shinytest2}</a>.</p>
<!-- source: code/tests/testthat/test-app.R -->
<pre>test_that(&quot;two sessions never share a client&quot;, {
 withr::local_envvar(GEMINI_API_KEY = &quot;test-key&quot;)
 # One app, as on a server: both sessions run inside it. Given a folder,
 # testServer() builds a fresh app each time, which would hide a shared client.
 app &lt;- shiny::shinyAppDir(test_path(&quot;..&quot;, &quot;..&quot;))
 seen &lt;- new.env()
 for (i in 1:2) {
 shiny::testServer(app, assign(paste0(&quot;client&quot;, i), chat$client, envir = seen))
 }
 expect_false(identical(seen$client1, seen$client2))
})
</pre><h2 id="showing-the-work">Showing the work</h2>
<p>{shinychat} shows the tool calls as rows above the answer. Each row takes its title from the <code>tool_annotations(title = …)</code> we gave the tool in part 1, such as “Counting subjects” or “Summarising adverse events”. Opening a row shows what the agent asked for and what came back, so a trial team can check where a number came from.</p>
<p>By default, calls to the same tool share one row, which lists each call when you open it. Set <code>tool_grouping = &quot;none&quot;</code> in <code>chat_ui()</code> to give every call its own row.</p>
<p>The raw results are JSON, which suits the model but is hard for people to read. So <code>with_display()</code> wraps each tool, and people see a readable table while the model still gets JSON:</p>
<!-- source: code/R/display.R, with_display() -->
<pre>with_display &lt;- function(tools) {
 lapply(tools, function(tool) {
 wrap_tool(tool, function(run, args) {
 value &lt;- run()
 # The display is for people. If it can&#39;t be built, the model still
 # gets its answer.
 markdown &lt;- tryCatch(result_markdown(value), error = function(e) NULL)
 if (is.null(markdown)) return(value)
 ellmer::ContentToolResult(
 value = value,
 extra = list(display = shinychat::tool_result_display(markdown = markdown))
 )
 })
 })
}
</pre><p>Two helpers do the work. The first, <code>wrap_tool()</code> in <code>R/wrap.R</code>, copies a tool so that every call goes through a function we give it. Inside that function, <code>run()</code> calls the original tool and <code>args</code> holds the arguments the tool will run with, including any defaults the model left out.</p>
<p>The second, <code>result_markdown()</code> in <code>R/display.R</code>, turns the JSON into a table or a list. We use <code>wrap_tool()</code> again in part 4, for the audit log.</p>
<p>Not every tool returns JSON. The handover tool in part 3 returns a plain sentence, and our first version of the display broke on it, taking the whole tool call down too. It now falls back to plain text.</p>
<aside class="advert">
<p>
Do you require help building a Shiny app? Would you like someone to take over the maintenance burden? If so, check out our <a href="https://www.jumpingrivers.com/consultancy/shiny-dash-flask-dashboard-consultancy/?utm_source=blog&#038;utm_medium=banner&#038;utm_campaign=2026-agents-in-production-shiny" rel="nofollow" target="_blank">Shiny and Dash</a> services.
</p>
</aside>
<h2 id="no-file-uploads">No file uploads</h2>
<p>By default, <code>chat_server()</code> allows file uploads. People can attach a CSV file, a PDF or an image, and it goes straight to the model. That undoes part 1’s design. The model would read raw data itself instead of asking our tested tools, and the whole file would go to the provider, when a question only needs a few numbers.</p>
<p>Setting <code>allow_attachments = FALSE</code> in <code>chat_ui()</code> only hides the upload button, and anyone can send a request without using the page. So the server checks every request as well. It refuses any user turn that holds more than typed text and tool results ({ellmer} sends tool results back as user turns).</p>
<p>Text files need one more check. {shinychat} sends a CSV or TXT file as plain text in a <code>&lt;file-attachment&gt;</code> wrapper, so the check refuses that too. {ellmer} builds turns and their contents as S7 objects, so the code tests their class with <code>S7::S7_inherits()</code> and reads their fields with <code>@</code>:</p>
<!-- source: code/R/attachments.R, attachment_check() and refuse_attachments() -->
<pre>attachment_check &lt;- function() {
 allowed &lt;- function(content) {
 if (S7::S7_inherits(content, ellmer::ContentText)) {
 # shinychat sends a text file (CSV, TXT, JSON and so on) as text in a
 # &lt;file-attachment&gt; wrapper, so text can be a file too.
 return(!grepl(&quot;^\\s*&lt;file-attachment &quot;, content@text))
 }
 S7::S7_inherits(content, ellmer::ContentToolResult)
 }
 function(turns) {
 for (turn in turns) {
 if (S7::S7_inherits(turn, ellmer::UserTurn) &#038;&#038; !all(vapply(turn@contents, allowed, logical(1)))) {
 rlang::abort(
 &quot;Files can&#39;t be sent to this assistant. Ask your question as text.&quot;,
 class = &quot;trial_attachment_refused&quot;
 )
 }
 }
 }
}

refuse_attachments &lt;- function(chat) {
 chat$on_request_start(attachment_check())
}
</pre><p>The check can’t stop someone pasting a file’s contents into the question, because no check can tell pasted data from a question. So what limits the risk is who can open the app and which provider you send text to, and part 4 covers both.</p>
<h2 id="when-the-model-is-down">When the model is down</h2>
<p>As we saw in <a href="https://www.jumpingrivers.com/blog/agents-in-production-first-agent/" rel="nofollow" target="_blank">part 1</a>, Gemini’s free tier often answers “503: high demand” or “429: quota exceeded”. When the provider is busy or rate-limited, {ellmer} tries each request up to three times, waiting longer each time, and <code>options(ellmer_max_tries = 5)</code> allows more.</p>
<p>If the last try fails, <code>chat$last_error()</code> fires. The chat shows {shinychat}’s generic line, “An error occurred. Please try again or contact the app author.”, and our notification tells the user what to do. Our <a href="https://www.jumpingrivers.com/blog/ellmer-dynamic-alt-text/" rel="nofollow" target="_blank">post on dynamic alt text</a> shows the same idea outside a chat app, with <code>tryCatch()</code> and a fallback.</p>
<h2 id="two-users-at-once">Two users at once</h2>
<p>R does one thing at a time. If one person’s 20-second question blocked the R process, everyone else’s app would freeze until it finished.</p>
<p>{shinychat}’s <code>chat_server()</code> avoids that. It streams each answer with {ellmer}’s asynchronous <code>stream_async()</code>, inside a Shiny <code>ExtendedTask</code>, so R keeps serving other sessions while one waits for the model. Tool calls still run one at a time in the shared process, so keep your tools quick. Ours take milliseconds.</p>
<figure class="w600">
<figcaption>Waiting for the model doesn't block R, so a second person's question starts straight away.</figcaption>
<a href="https://www.jumpingrivers.com/blog/agents-in-production-shiny/assets/two-sessions-one-process.svg" rel="nofollow" target="_blank"><img src="https://www.jumpingrivers.com/blog/agents-in-production-shiny/assets/two-sessions-one-process.svg" alt="Timeline diagram. Browser 1 sends a question, then waits while the model works and the answer streams in. Partway through, browser 2 sends its own question and waits too. Below them, the shared R process only has short jobs to do, such as running one tool call for each session, and is free the rest of the time. A note says tool calls run in R one at a time, so keep them quick, and that the waiting happens at the model provider."></a>
</figure>
<p>To check by hand, open two browser windows. Ask in the first and, while its answer is still arriving, ask in the second. The second answer should start straight away. Our <a href="https://www.jumpingrivers.com/blog/sip2025-shiny-conference-summary/" rel="nofollow" target="_blank">Shiny in Production 2025 summary</a> has more on asynchronous Shiny.</p>
<h2 id="deploying-to-posit-connect">Deploying to Posit Connect</h2>
<p><a href="https://posit.co/products/enterprise/connect/" rel="nofollow" target="_blank">Posit Connect</a> runs Shiny apps for a team, with sign-in and access control.</p>
<p><strong>Treat this first deploy as a private test.</strong> The free tier is fine for this public data, but not for your own patient or confidential data. And an app offered to users in the UK, the EEA or Switzerland must use Google’s paid service, as part 4 explains.</p>
<p>This version also lets anyone who can open the app see patient records, and it keeps no log. Parts 3 and 4 fix both.</p>
<p>You register your Connect account once, and the code’s README has the steps. After that, deploying is one call from the project folder. The <code>appFiles</code> argument lists what goes into the bundle: the app, the <code>R/</code> folder and the system prompt. Tests and local files stay behind.</p>
<pre>rsconnect::deployApp(
 &quot;.&quot;,
 appFiles = c(&quot;app.R&quot;, list.files(&quot;R&quot;, full.names = TRUE), &quot;prompts/system.md&quot;),
 appName = &quot;trial-safety-assistant&quot;,
 envVars = &quot;GEMINI_API_KEY&quot;
)
</pre><p>The API key never goes in the code or the bundle. Instead, <code>envVars</code> copies it from your R session, which read it from <code>.Renviron</code>, into the app’s <strong>Vars</strong> panel in Connect, over an encrypted connection, before the app first starts. Every later deploy copies it again, so change the key in your <code>.Renviron</code> as well as in the panel.</p>
<p>Beyond a private test, use a key from your organisation’s own account, under its paid terms, rather than your personal one.</p>
<p>To deploy on every push instead, see Keith’s post on <a href="https://www.jumpingrivers.com/blog/github-actions-deployment-to-posit-connect-cloud/" rel="nofollow" target="_blank">GitHub Actions deployments to Posit Connect Cloud</a>, or use git-backed publishing on your own Connect server.</p>
<p>We deployed it to our own Connect server, version 2026.06.1, and repeated the two-window check there. Each window kept its own conversation.</p>
<p>The bundle held only the nine files listed. Because the folder has an <code>renv.lock</code>, though, {rsconnect} took the package list from it, so Connect installed all 100 packages in the lockfile, including those only the tests use. That makes the first deploy slower, but it does no harm.</p>
<aside>
<h3 id="in-python">In Python</h3>
<p>Shiny for Python has a chat component that streams from chatlas. Passing <code>content=&quot;all&quot;</code> streams the tool calls too, so they show in the chat as they do in the R app:</p>
<pre>from pathlib import Path

from chatlas import ChatGoogle
from shiny.express import ui

# Shiny Express runs this whole file once per session,
# so each user gets their own client.
client = ChatGoogle(
 model=&quot;gemini-3.7-flash&quot;,
 system_prompt=Path(&quot;prompts/system.md&quot;).read_text(encoding=&quot;utf-8&quot;),
)
# client.register_tool(...) for each tool, as in part 1

chat = ui.Chat(id=&quot;chat&quot;)
chat.ui()

@chat.on_user_submit
async def _(user_input: str):
 response = await client.stream_async(user_input, content=&quot;all&quot;)
 await chat.append_message_stream(response)
</pre></aside>
<h2 id="summary">Summary</h2>
<p>The agent from part 1 is now an app that several people can use at once. Each person gets their own conversation, and the tool calls are visible. File attachments are refused, a failed question tells the user what to do next, and the app runs on Posit Connect.</p>
<p>Before you share it, run through Tim’s <a href="https://www.jumpingrivers.com/blog/five-preflight-checks-dashboard/" rel="nofollow" target="_blank">five pre-flight checks for your dashboard</a>. They apply to a chat app too.</p>
<p>Next time, we take on the question every trial team asks first: can you trust it? We’ll test the tools, evaluate the agent and check where every number in an answer came from. We’ll also let only reviewers open a patient record, and have them confirm each one.</p>
<p>All the code for this part, including the app and its tests, is in <a href="https://github.com/jumpingrivers/blog/tree/main/blogs/agents-in-production/part-2" rel="nofollow" target="_blank">our blog repository</a>.</p>
<p>If you’re building AI tools with Shiny, come and talk about it at <a href="https://ai-in-production.jumpingrivers.com/" rel="nofollow" target="_blank">AI in Production 2027</a>, where the call for talks is open until 29 January 2027. Our <a href="https://www.jumpingrivers.com/training/course/llm-applications-r-python-shiny-rag-ai/" rel="nofollow" target="_blank">LLM-Driven Applications with R and Python</a> course also covers adding a chat to a Shiny app.</p>
<p>
For updates and revisions to this article, see the <a href = "https://www.jumpingrivers.com/blog/agents-in-production-shiny/">original post</a>
</p>
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.jumpingrivers.com/blog/agents-in-production-shiny/"> The Jumping Rivers Blog</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/agents-in-production-build-an-ai-chat-app-with-shiny-and-shinychat/">Agents in Production: Build an AI Chat App with Shiny and {shinychat}</a>]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">404752</post-id>	</item>
		<item>
		<title>Growing the BiocAsia community</title>
		<link>https://www.r-bloggers.com/2026/11/growing-the-biocasia-community/</link>
		
		<dc:creator><![CDATA[Maria Doyle]]></dc:creator>
		<pubDate>Tue, 10 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://blog.bioconductor.org/posts/2026-11-10-growing-biocasia-community/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>With BiocAsia 2026 coming up in Melbourne, community members have been organising seminars and exploring ways to take part locally.<br />
More than 200 people joined the latest BiocAsia seminar on 13 October 2026. The session ran for two and a half ho...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/growing-the-biocasia-community/">Growing the BiocAsia community</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://blog.bioconductor.org/posts/2026-11-10-growing-biocasia-community/"> Bioconductor community blog</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 





<p>With <a href="https://biocasia2026.bioconductor.org/" rel="nofollow" target="_blank">BiocAsia 2026</a> coming up in Melbourne, community members have been organising seminars and exploring ways to take part locally.</p>
<p>More than 200 people joined the latest <a href="https://biocasia2026.bioconductor.org/seminars/" rel="nofollow" target="_blank">BiocAsia seminar</a> on 13 October 2026. The session ran for two and a half hours, with discussion continuing well beyond the planned finish. It was the first seminar in the series hosted from China, with Mandarin as the main language. Shixiang Wang spoke about building an open-source ecosystem around UCSC Xena and visualisation tools, and Zhougeng Xu shared his experience developing open-source tools for AI in science.</p>
<p>The seminar was organised by Zuguang Gu, a member of the <a href="https://bioconductor.org/about/community-advisory-board/" rel="nofollow" target="_blank">Bioconductor Community Advisory Board</a> and an active contributor to BiocAsia based in China. The next seminar will take place in December.</p>
<div style="text-align: center;">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/blog.bioconductor.org/posts/2026-11-10-growing-biocasia-community/biocasia-seminar.jpg?w=578&#038;ssl=1" class="img-fluid figure-img" alt="Screenshot of the online seminar during Zhougeng Xu’s talk." data-recalc-dims="1"></p>
<figcaption>Zhougeng Xu’s presentation at the October 2026 BiocAsia seminar</figcaption>
</figure>
</div>
</div>
<p>Alongside the seminars, Jyoti Bhogal and Malvika Kharbanda have been discussing watch party options in India for BiocAsia 2026. The aim is to give people who cannot travel to Melbourne a chance to take part in the conference and meet others nearby.</p>
<p>Join us at <a href="https://biocasia2026.bioconductor.org/" rel="nofollow" target="_blank">BiocAsia 2026</a> on <strong>19 and 20 November at RMIT City Campus, Melbourne</strong>, for workshops, talks and a chance to meet others working with R and Bioconductor. <a href="https://biocasia2026.bioconductor.org/registration/" rel="nofollow" target="_blank"><strong>Registration is open</strong></a>, and we’d love to see you there.</p>
<p>To keep in touch between events, <a href="https://community-bioc.zulipchat.com/join/cr4isltci5v232hkkrhti656/" rel="nofollow" target="_blank">join the <strong>BiocAsia channel on Bioconductor Zulip</strong></a>. It’s a place to hear about future seminars, discuss watch parties and share ideas for other activities.</p>
<p>A big thank you to the speakers, organisers and everyone giving their time to make these activities happen.</p>



<p>
© 2026 Bioconductor. Content is published under <a href="https://creativecommons.org/licenses/by/4.0/" rel="nofollow" target="_blank">Creative Commons CC-BY-4.0 License</a> for the text and <a href="https://opensource.org/licenses/BSD-3-Clause" rel="nofollow" target="_blank">BSD 3-Clause License</a> for any code. | <a href="https://www.r-bloggers.com/" rel="nofollow" target="_blank">R-Bloggers</a>
</p> 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.bioconductor.org/posts/2026-11-10-growing-biocasia-community/"> Bioconductor community blog</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/growing-the-biocasia-community/">Growing the BiocAsia community</a>]]></content:encoded>
					
		
		<enclosure url="https://blog.bioconductor.org/posts/2026-11-10-growing-biocasia-community/biocasia-seminar.jpg" length="0" type="image/jpeg" />

		<post-id xmlns="com-wordpress:feed-additions:1">404767</post-id>	</item>
		<item>
		<title>Using R to make an “informational video”</title>
		<link>https://www.r-bloggers.com/2026/11/using-r-to-make-an-informational-video/</link>
		
		<dc:creator><![CDATA[nsaunders]]></dc:creator>
		<pubDate>Mon, 09 Nov 2026 00:56:56 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">http://nsaunders.wordpress.com/?p=5750</guid>

					<description><![CDATA[<p>I volunteer at my local junior AFL club and my role as club registrar grants access to a sports management platform called PlayHQ. It’s a rich source of data so for the end of the 2026 season, I thought it would be fun to use some of that data to ...</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/using-r-to-make-an-informational-video/">Using R to make an “informational video”</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://nsaunders.wordpress.com/2026/11/09/using-r-to-make-an-informational-video/"> R – What You&#039;re Doing Is Rather Desperate</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>

<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<div class="jetpack-video-wrapper"><iframe class="youtube-player" title="YouTube video player" width="450" src="https://www.youtube.com/embed/Q4rVdOJ904Y?version=3&#038;rel=1&#038;%23038;showsearch=0&#038;%23038;showinfo=1&#038;%23038;iv_load_policy=1&#038;%23038;fs=1&#038;%23038;hl=en&#038;%23038;autohide=2&#038;%23038;wmode=transparent" allowfullscreen="true" style="border:0;" sandbox="allow-scripts allow-same-origin allow-popups allow-presentation allow-popups-to-escape-sandbox"></iframe></div>
</div></figure>



<p class="wp-block-paragraph">I volunteer at my local junior AFL club and my role as club registrar grants access to a sports management platform called PlayHQ. It’s a rich source of data so for the end of the 2026 season, I thought it would be fun to use some of that data to create a season summary in an informational video format. You can see the result in the video, above.</p>



<span id="more-5750"></span>



<p class="wp-block-paragraph">This post is not a how-to, just a brief summary of the tools that I used.</p>



<p class="wp-block-paragraph">The input data: CSV files downloaded from the available reports in PlayHQ. These include data describing player registration, games played and venues.</p>



<p class="wp-block-paragraph">A lot of R users seem to be moving towards the Positron IDE and Quarto to generate presentation material. I’m still using RStudio and RMarkdown, simply because that’s what I know best right now. The key packages for this project: <em>ggplot2</em> of course; <em>waffle</em> to show player counts; <em>sf</em>, <em>leaflet</em>, <em>geosphere</em>, <em>tidyterra</em>, <em>osrm</em> and <em>maptiles</em> for geographical data; <em>wordcloud</em> for, well, word clouds.</p>



<p class="wp-block-paragraph">I used the RMarkdown file to sketch out each slide and generate one high-quality graphic per slide in PNG format, or in some cases as an animated GIF. Next, I used Keynote to create the final slides from the imported PNG files. I’ve had Keynote sitting largely unopened on every Mac that I’ve owned for years – turns out that it’s pretty good – not least because you can export the final product straight to video. I used a custom resolution of 1170×2532 to match my (iPhone 13) screen, which seems to work OK on most current phone screens.</p>



<p class="wp-block-paragraph">There’s plenty of free-to-use cheesy corporate music online to choose from, then it’s just a case of fiddling with the slide transitions to sync the music and any animations. Getting the timing right for slides that contain animated GIFs can be a bit tricky – quite honestly I don’t recall what worked best, other than clicking on the item, selecting the “Movie” tab in the right sidebar menu and “adjusting until it works”.</p>



<p class="wp-block-paragraph">Once the run-through in Keynote looks and sounds good, export to video and it’s ready to share.</p>

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://nsaunders.wordpress.com/2026/11/09/using-r-to-make-an-informational-video/"> R – What You&#039;re Doing Is Rather Desperate</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/using-r-to-make-an-informational-video/">Using R to make an “informational video”</a>]]></content:encoded>
					
		
		<enclosure url="https://0.gravatar.com/avatar/0ace56430001dc0f2b92cfca798b8b6469cf7fc2c2878945baf9672a3d46e3f6?s=96&#038;d=https://0.gravatar.com/avatar/ad516503a11cd5ca435acc9bb6523536?s=96" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404748</post-id>	</item>
		<item>
		<title>Using caretS3 with ahead::ctxthetaf for Time Series Forecasting</title>
		<link>https://www.r-bloggers.com/2026/11/using-carets3-with-aheadctxthetaf-for-time-series-forecasting/</link>
		
		<dc:creator><![CDATA[T. Moudiki]]></dc:creator>
		<pubDate>Sat, 07 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://thierrymoudiki.github.io//blog/2026/11/07/r/ctxthetaf-caretS3</guid>

					<description><![CDATA[<p>A demonstration of using caretS3 models for time series forecasting with ahead::ctxthetaf.</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/using-carets3-with-aheadctxthetaf-for-time-series-forecasting/">Using caretS3 with ahead::ctxthetaf for Time Series Forecasting</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://thierrymoudiki.github.io//blog/2026/11/07/r/ctxthetaf-caretS3"> T. Moudiki's Webpage - R</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<p><a href="https://techtonique.r-universe.dev/ahead/doc/manual.html#ctxthetaf" rel="nofollow" target="_blank"><code>ahead::ctxthetaf</code></a> has no more contextual information about the time series (<a href="https://techtonique.r-universe.dev/ahead/doc/manual.html#contextridge2f" rel="nofollow" target="_blank">ahead::contextridge2f</a> still does), but it still works, and can still be used notably with <code>caretS3</code> (a <code>caret</code> derivative introduced in <a href="https://thierrymoudiki.github.io/blog/2026/10/25/r/caretS3-ahead" rel="nofollow" target="_blank">https://thierrymoudiki.github.io/blog/2026/10/25/r/caretS3-ahead</a>) to forecast time series data. The following code demonstrates how to use <code>caretS3</code> models for time series forecasting with <code>ahead::ctxthetaf</code>.</p>

<pre>list_models &lt;- caret::modelLookup()[caret::modelLookup()[,&quot;forReg&quot;]==TRUE, 1]

n_models &lt;- length(list_models)

pb &lt;- utils::txtProgressBar(min = 0, max = n_models, style = 3)

for (i in seq_len(n_models))
{
  ts_model &lt;- try(ahead::ctxthetaf(y=fdeaths, 
                                   theta = 1,
                                   h=25L, 
                                   fit_func = function (x, y) caretS3::caretS3(x, y, 
                                                                               method=list_models[i])), 
                  silent = TRUE)
  if (class(ts_model) != &quot;try-error&quot;)
  {
    plot(ts_model) 
  } 
  utils::setTxtProgressBar(pb, i)
}

close(pb)
</pre>


<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://thierrymoudiki.github.io//blog/2026/11/07/r/ctxthetaf-caretS3"> T. Moudiki's Webpage - R</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/using-carets3-with-aheadctxthetaf-for-time-series-forecasting/">Using caretS3 with ahead::ctxthetaf for Time Series Forecasting</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404732</post-id>	</item>
		<item>
		<title>Free AI and Data Courses with 365 Data Science: Unlimited Access until Nov. 20</title>
		<link>https://www.r-bloggers.com/2026/11/free-ai-and-data-courses-with-365-data-science-unlimited-access-until-nov-20/</link>
		
		<dc:creator><![CDATA[neddd]]></dc:creator>
		<pubDate>Fri, 06 Nov 2026 12:42:11 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://r-posts.com/?p=19631</guid>

					<description><![CDATA[<p>Free AI and Data Courses with 365 Data Science: Unlimited Access until Nov. 20 365 Data Science opens its doors this November 100% for free! From November 5 to November 20, 2026, enjoy unlimited access to the entire platform, including a wide range of R programming courses and projects. This offer features expert-led courses, practical … Continue reading ...</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/free-ai-and-data-courses-with-365-data-science-unlimited-access-until-nov-20/">Free AI and Data Courses with 365 Data Science: Unlimited Access until Nov. 20</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="http://r-posts.com/free-ai-and-data-courses-with-365-data-science-unlimited-access-until-nov-20/"> R-posts.com</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<p><strong>Free AI and Data Courses with 365 Data Science: Unlimited Access until Nov. 20</strong></p>
<p><u><a href="https://bit.ly/48m6TnO" rel="nofollow" target="_blank">365 Data Science opens its doors this November 100% for free!</a></u></p>
<p>From November 5 to November 20, 2026, enjoy unlimited access to the entire platform, including a wide range of <a href="https://bit.ly/48m6TnO" rel="nofollow" target="_blank">R programming</a> courses and projects.</p>
<p>This offer features expert-led courses, practical projects, and interactive exercises covering diverse topics in AI and data science. With flexible 24/7 access, learners can study at their own pace, making it an ideal opportunity to grow and advance in these fields.</p>
<p><img loading="lazy" fetchpriority="high" decoding="async" src="https://i2.wp.com/r-posts.com/wp-content/uploads/2026/11/R-posts-Picture-1-450x236.jpg?resize=450%2C236" alt="" width="450" height="236" class="alignnone size-medium wp-image-19629" srcset_temp="https://i2.wp.com/r-posts.com/wp-content/uploads/2026/11/R-posts-Picture-1-450x236.jpg?resize=450%2C236 450w, http://r-posts.com/wp-content/uploads/2026/11/R-posts-Picture-1.jpg 468w" sizes="(max-width: 450px) 100vw, 450px" data-recalc-dims="1" /></p>
<p> </p>
<p><strong>The Initiative’s Sixth and Biggest Year</strong></p>
<p>Now in its sixth year, 365 Data Science reaffirms its dedication to delivering accessible, high-quality education through its annual Free Access Initiative, originally launched in 2020 during the global pandemic. CEO Ned Krastev emphasizes the accelerating demand for AI-driven skills, stating, <em>“The AI and data landscape is evolving faster than ever, creating extraordinary opportunities for career growth for those ready to embrace new technologies.”</em></p>
<p>The initiative’s reach has grown remarkably: 2025 marked its most successful edition yet, with over 220,000 unique learners from 216 countries participating, accumulating 7.2 million minutes of learning and earning more than 36,000 certificates.</p>
<p>Krastev adds, <em>“Artificial intelligence is reshaping industries at an unprecedented pace. Understanding how AI systems function</em><em> and how to build, deploy, and integrate them</em><em> has become vital for anyone pursuing a data-driven career. At 365 Data Science, our mission is to close that gap by equipping learners not only with data literacy but also with hands-on expertise in AI engineering and intelligent agents: the skills that will define the next generation of tech professionals.”</em></p>
<p><img loading="lazy" decoding="async" src="https://i0.wp.com/r-posts.com/wp-content/uploads/2026/11/R-posts-Picture-2-450x225.png?resize=450%2C225" alt="" width="450" height="225" class="alignnone size-medium wp-image-19630" srcset_temp="https://i0.wp.com/r-posts.com/wp-content/uploads/2026/11/R-posts-Picture-2-450x225.png?resize=450%2C225 450w, http://r-posts.com/wp-content/uploads/2026/11/R-posts-Picture-2.png 624w" sizes="(max-width: 450px) 100vw, 450px" data-recalc-dims="1" /></p>
<p>365 Data Science empowers learners to move beyond traditional analytics and step confidently into the era of AI engineering and intelligent agents. The platform equips users with the skills to design, deploy, and collaborate with AI systems capable of reasoning, planning, and acting autonomously.</p>
<p><strong>What’s Included</strong></p>
<p>During this special period, learners <a href="https://bit.ly/48m6TnO" rel="nofollow" target="_blank">gain unlimited access to the entire 365 Data Science platform</a>, a comprehensive destination for mastering data and AI. The platform offers 134 expert-led courses, covering everything from fundamental data skills to advanced topics in machine learning, AI, and AI engineering.</p>
<p>Participants can apply their knowledge through real-world projects that mirror practical work scenarios, gaining hands-on experience in solving meaningful problems. Newly added interactive exercises and guided challenges deepen understanding and strengthen key concepts.</p>
<p>To support career growth, 365 Data Science also provides structured, career-focused learning paths guiding learners step by step from beginner to job-ready professional with a clear roadmap to success in today’s AI-driven world.</p>
<p><strong>Certifications that Open Doors</strong></p>
<p>In today’s fast-changing job market, recognized certifications are essential to stand out. Through this free access initiative, 365 Data Science offers learners the chance to earn industry-recognized certificates at no cost.</p>
<p>These credentials validate practical expertise in data analytics, AI, and machine learning, enhancing employability and building credibility with employers worldwide. By offering verifiable, career-boosting certifications, the initiative bridges the gap between learning and professional achievement by showcasing real-world competence.</p>
<p><strong>R Programming: A Core Focus</strong></p>
<p>365 Data Science acknowledges <strong>the central role of R</strong> in modern data analysis and visualization, reflected in its carefully curated course selection:</p>
<ul>
	<li><strong>Introduction to R Programming:</strong> A beginner-friendly course that builds a strong foundation in R syntax, data structures, and essential operations—perfect for kick-starting your R journey.</li>
	<li><strong>The Complete Data Visualization Course with Python, R, Tableau, and Excel:</strong> A comprehensive program that highlights R’s powerful visualization tools while exploring multiple platforms for data storytelling.</li>
	<li><strong>Data-Driven Business Growth:</strong> Learn how to apply R to real-world business cases, adopting a growth mindset to generate meaningful insights and drive measurable impact.</li>
</ul>
<p>Additionally, learners can explore a variety of complementary courses designed to deepen their analytical toolkit, covering Statistics, Probability, Statistical Tests in Sales and Marketing, Data Preprocessing, and more—helping you master both R and other key programming languages in data science.</p>
<p><strong>Don’t Miss this Opportunity</strong></p>
<p>Free access runs from November 5 to November 20, 2026.</p>
<p>In a world increasingly powered by data and artificial intelligence, staying ahead of the curve is essential. This three-week open-access period from 365 Data Science offers a rare opportunity to invest in your future: whether you’re just getting started, switching careers, or advancing your expertise in AI and data.</p>
<p>Don’t miss your chance to build in-demand skills, earn industry-recognized certificates, and take the next step toward a successful career in data science and AI engineering.</p>
<p><em>The future belongs to those who prepare for it today: </em><a href="https://bit.ly/48m6TnO" rel="nofollow" target="_blank"><em>start your journey for free with 365 Data Science.</em></a></p>
<p> </p><hr style="border-top: black solid 1px" /><a href="http://r-posts.com/free-ai-and-data-courses-with-365-data-science-unlimited-access-until-nov-20/" rel="nofollow" target="_blank">Free AI and Data Courses with 365 Data Science: Unlimited Access until Nov. 20</a> was first posted on November 6, 2026 at 12:42 pm.<br />
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-posts.com/free-ai-and-data-courses-with-365-data-science-unlimited-access-until-nov-20/"> R-posts.com</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/free-ai-and-data-courses-with-365-data-science-unlimited-access-until-nov-20/">Free AI and Data Courses with 365 Data Science: Unlimited Access until Nov. 20</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404706</post-id>	</item>
		<item>
		<title>Nuclei, five years later: what a mean Dice hides</title>
		<link>https://www.r-bloggers.com/2026/11/nuclei-five-years-later-what-a-mean-dice-hides/</link>
		
		<dc:creator><![CDATA[Michał Maj]]></dc:creator>
		<pubDate>Fri, 06 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>In 2020 I trained a U-Net on the 2018 Data Science Bowl nuclei with the first version of this package, watched a Dice score climb during training, and finished the post with four predicted masks on the competition’s test images. Those images ha...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/nuclei-five-years-later-what-a-mean-dice-hides/">Nuclei, five years later: what a mean Dice hides</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/"> Data Science Guts</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 






<p>In 2020 I trained a U-Net on the <a href="https://www.kaggle.com/c/data-science-bowl-2018" rel="nofollow" target="_blank">2018 Data Science Bowl</a> nuclei with the first version of this package, watched a Dice score climb during training, and finished the post with four predicted masks on the competition’s test images. Those images have no masks. The pictures looked good, and I had no way of knowing where the model was wrong &#8211; so I never asked.</p>
<p>This post asks. Nothing in it trains: the model is the published <code>dsbowl-unet</code> from the <a href="https://datascienceguts.com/posts/2026-11-03-rplatypus/" rel="nofollow" target="_blank">first post of the series</a>, scored on the images it was held out from. Scoring all of them took about twelve seconds on a processor.</p>
<section id="the-images-it-was-held-out-from" class="level2">
<h2 class="anchored" data-anchor-id="the-images-it-was-held-out-from">The images it was held out from</h2>
<p>The data is <a href="https://bbbc.broadinstitute.org/BBBC038" rel="nofollow" target="_blank">BBBC038v1</a> from the Broad Institute &#8211; the Data Science Bowl’s images under a CC0 waiver. Its <code>stage1_train</code> holds 670 annotated images, one directory each, with one mask file per nucleus.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-1-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-1-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-1-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-1-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-1-1" class="tab-pane active" aria-labelledby="tabset-1-1-tab">
<div class="cell">
<pre>data_dir &lt;- &quot;stage1_train&quot;   # BBBC038v1, unzipped</pre>
</div>
</div>
<div id="tabset-1-2" class="tab-pane" aria-labelledby="tabset-1-2-tab">
<pre>data_dir = &quot;stage1_train&quot;   # BBBC038v1, unzipped</pre>
</div>
</div>
</div>
<p>The weights were measured on an 80/20 split of all 670 at seed 1, so that split can be made again. <code>split_dataset()</code> copies nothing; it writes CSVs listing the files:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-2-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-2-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-2-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-2-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-2-1" class="tab-pane active" aria-labelledby="tabset-2-1-tab">
<div class="cell">
<pre>library(rplatypus)

split &lt;- split_dataset(data_dir, tempfile(&quot;splits&quot;),
                       fractions = c(0.8, 0.2), seed = 1)
split
#&gt; platypus split
#&gt;   train        536 samples
#&gt;   validation   134 samples
#&gt;   every sample its own group
#&gt;   seed 1 - the same inputs give this split again</pre>
</div>
</div>
<div id="tabset-2-2" class="tab-pane" aria-labelledby="tabset-2-2-tab">
<pre>import tempfile
import pyplatypus as pp

split = pp.split_dataset(data_dir, tempfile.mkdtemp(),
                         fractions=(0.8, 0.2), seed=1)
split[&quot;samples&quot;]
#&gt; {'train': 536, 'validation': 134, 'test': 0}</pre>
</div>
</div>
</div>
<p>Making the split again matters more than it looks. My own copy of this dataset has a directory called <code>stage1_validation</code> with exactly 134 images in it - the same count - and 101 of them are in the training part of the split the weights were measured on. Scored there, the model would be graded partly on images it learnt from, and the count would never tell you.</p>
</section>
<section id="one-number" class="level2">
<h2 class="anchored" data-anchor-id="one-number">One number</h2>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-3-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-3-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-3-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-3-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-3-1" class="tab-pane active" aria-labelledby="tabset-3-1-tab">
<div class="cell">
<pre>spec &lt;- platypus_spec(
  data = segmentation_data(split, colormap = binary_colormap),
  models = list(
    u_net(&quot;nuclei&quot;, input_shape = c(256, 256), blocks = 4, filters = 16,
          weights = &quot;dsbowl-unet&quot;, fit = FALSE,
          metrics = list(metric_dice(include_background = FALSE)))
  )
)
fit &lt;- platypus_fit(spec, verbose = FALSE)

cases &lt;- evaluate_cases(fit)
summary(cases)
#&gt;   metric   n      mean         sd    median       min       max
#&gt; 1   dice 134 0.9205236 0.05322748 0.9313503 0.7239583 0.9860661
#&gt;                                                         worst_case
#&gt; 1 7f38885521586fc6011bef1314a9fb2aa1e4935bd581b2991e1d963395eab770</pre>
</div>
</div>
<div id="tabset-3-2" class="tab-pane" aria-labelledby="tabset-3-2-tab">
<pre>engine = pp.build_engine(pp.from_dict({
    &quot;task&quot;: &quot;semantic_segmentation&quot;,
    &quot;data&quot;: {
        &quot;train_path&quot;: str(split[&quot;paths&quot;][&quot;train&quot;]),
        &quot;validation_path&quot;: str(split[&quot;paths&quot;][&quot;validation&quot;]),
        &quot;colormap&quot;: [[0, 0, 0], [255, 255, 255]],
        &quot;mode&quot;: &quot;config_file&quot;,
    },
    &quot;models&quot;: [{
        &quot;name&quot;: &quot;nuclei&quot;,
        &quot;architecture&quot;: &quot;u_net&quot;,
        &quot;input_shape&quot;: [256, 256],
        &quot;blocks&quot;: 4,
        &quot;filters&quot;: 16,
        &quot;weights&quot;: &quot;dsbowl-unet&quot;,
        &quot;fit&quot;: False,
        &quot;metrics&quot;: [{&quot;name&quot;: &quot;dice&quot;, &quot;include_background&quot;: False}],
    }],
}))
engine.fit()

cases = engine.evaluate_cases(&quot;nuclei&quot;)
pp.summarise_cases(cases)
#&gt; [{'metric': 'dice', 'n': 134, 'mean': 0.9205236150257623,
#&gt;   'sd': 0.05322747958922138, 'median': 0.9313502907752991,
#&gt;   'min': 0.7239583134651184, 'max': 0.9860661029815674,
#&gt;   'worst_case':
#&gt;     '7f38885521586fc6011bef1314a9fb2aa1e4935bd581b2991e1d963395eab770'}]</pre>
</div>
</div>
</div>
<p>A mean Dice of 0.9205 and a worst image at 0.7240 - the model card’s numbers to four digits, which is the check that this is the card’s split. <code>include_background = FALSE</code> keeps the background out of the average; with it in, a model that found no nuclei at all would still score around half.</p>
<p>0.92 is a good number, and it is true. The question is what it is an average <em>of</em>.</p>
</section>
<section id="one-row-per-image" class="level2">
<h2 class="anchored" data-anchor-id="one-row-per-image">One row per image</h2>
<p><code>evaluate()</code> gives one row per model. <code>evaluate_cases()</code> gives one row per image, and that is the table worth reading:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-4-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-4-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-4-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-4-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-4-1" class="tab-pane active" aria-labelledby="tabset-4-1-tab">
<div class="cell">
<pre>head(cases[order(cases$dice), ], 5)
#&gt; platypus scores by case (5 rows)
#&gt;                                                              case      dice
#&gt;  7f38885521586fc6011bef1314a9fb2aa1e4935bd581b2991e1d963395eab770 0.7239583
#&gt;  29ea4f6eb4545f43868a9b40a60000426bf8dfd9d062546656a37bd2a2aaf9ec 0.7531800
#&gt;  8f94a80b95a881d0efdec36affc915dca9609f4cba8134c4a91b219d418778aa 0.7541562
#&gt;  a6515d73077866808ad4cb837ecdac33612527b8a1041e82135e40fce2bb9380 0.7588775
#&gt;  2a1a294e21d76efd0399e4eb321b45f44f7510911acd92c988480195c5b4c812 0.7702851</pre>
</div>
</div>
<div id="tabset-4-2" class="tab-pane" aria-labelledby="tabset-4-2-tab">
<pre>worst = sorted(cases, key=lambda row: row[&quot;dice&quot;])[:5]
[(row[&quot;case&quot;][:16], round(row[&quot;dice&quot;], 4)) for row in worst]
#&gt; [('7f38885521586fc6', 0.724), ('29ea4f6eb4545f43', 0.7532),
#&gt;  ('8f94a80b95a881d0', 0.7542), ('a6515d7307786680', 0.7589),
#&gt;  ('2a1a294e21d76efd', 0.7703)]</pre>
</div>
</div>
</div>
<p>Those names say nothing, so give each image a label. The Data Science Bowl mixed three kinds of microscopy, and two numbers tell them apart well enough: whether the picture has colour, and whether its background is light or dark.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-5-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-5-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-5-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-5-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-5-1" class="tab-pane active" aria-labelledby="tabset-5-1-tab">
<div class="cell">
<pre>image_kind &lt;- function(case) {
  path &lt;- list.files(file.path(data_dir, case, &quot;images&quot;), full.names = TRUE)
  pixels &lt;- png::readPNG(path[1])[, , 1:3]
  colour &lt;- mean(abs(pixels[, , 1] - pixels[, , 2])) +
    mean(abs(pixels[, , 2] - pixels[, , 3]))
  if (colour &gt; 0.02) return(&quot;stained&quot;)
  if (mean(pixels) &gt; 0.5) &quot;brightfield&quot; else &quot;fluorescence&quot;
}
cases$kind &lt;- vapply(cases$case, image_kind, character(1))

aggregate(dice ~ kind, cases, function(x) {
  round(c(n = length(x), mean = mean(x), min = min(x)), 3)
})
#&gt;           kind  dice.n dice.mean dice.min
#&gt; 1  brightfield   3.000     0.749    0.724
#&gt; 2 fluorescence 105.000     0.935    0.753
#&gt; 3      stained  26.000     0.883    0.805</pre>
</div>
</div>
<div id="tabset-5-2" class="tab-pane" aria-labelledby="tabset-5-2-tab">
<pre>from pathlib import Path
import numpy as np
from PIL import Image

def image_kind(case):
    path = next((Path(data_dir) / case / &quot;images&quot;).glob(&quot;*.png&quot;))
    pixels = np.asarray(Image.open(path).convert(&quot;RGB&quot;)) / 255
    colour = (np.abs(pixels[..., 0] - pixels[..., 1]).mean()
              + np.abs(pixels[..., 1] - pixels[..., 2]).mean())
    if colour &gt; 0.02:
        return &quot;stained&quot;
    return &quot;brightfield&quot; if pixels.mean() &gt; 0.5 else &quot;fluorescence&quot;

for row in cases:
    row[&quot;kind&quot;] = image_kind(row[&quot;case&quot;])

kinds = sorted({row[&quot;kind&quot;] for row in cases})
dice = {k: [r[&quot;dice&quot;] for r in cases if r[&quot;kind&quot;] == k] for k in kinds}
{k: (len(d), round(float(np.mean(d)), 3), round(min(d), 3))
 for k, d in dice.items()}
#&gt; {'brightfield': (3, 0.749, 0.724), 'fluorescence': (105, 0.935, 0.753),
#&gt;  'stained': (26, 0.883, 0.805)}</pre>
</div>
</div>
</div>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-6-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-6-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-6-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-6-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-6-1" class="tab-pane active" aria-labelledby="tabset-6-1-tab">
<div class="cell">
<pre>library(ggplot2)
ggplot(cases, aes(dice, kind)) +
  geom_vline(xintercept = mean(cases$dice), linetype = &quot;dashed&quot;,
             colour = &quot;grey50&quot;) +
  geom_jitter(height = 0.15, width = 0, alpha = 0.7, size = 2) +
  labs(x = &quot;Dice, per image&quot;, y = NULL) +
  theme_minimal(base_size = 13)</pre>
</div>
</div>
<div id="tabset-6-2" class="tab-pane" aria-labelledby="tabset-6-2-tab">
<pre>import matplotlib.pyplot as plt

rng = np.random.default_rng(1)
fig, ax = plt.subplots(figsize=(7, 3.2))
ax.axvline(np.mean([r[&quot;dice&quot;] for r in cases]), linestyle=&quot;--&quot;, color=&quot;grey&quot;)
for i, k in enumerate(kinds):
    jitter = rng.uniform(-0.15, 0.15, len(dice[k]))
    ax.scatter(dice[k], i + jitter, alpha=0.7)
ax.set_yticks(range(len(kinds)), kinds)
ax.set_xlabel(&quot;Dice, per image&quot;)</pre>
</div>
</div>
</div>
<div class="cell">
<div class="cell-output-display">
<div id="fig-cases" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-cases-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://i2.wp.com/datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/index_files/figure-html/fig-cases-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-cases-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 1: Dice for each of the 134 held-out images. The dashed line is the mean.
</figcaption>
</figure>
</div>
</div>
</div>
<p>There are three brightfield images among the 134, and they score 0.72, 0.75 and 0.77 - three of the five worst. Fluorescence averages 0.93 over 105 images, and it is most of what the mean measures. Stained tissue sits between the two, at 0.88.</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-7-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-7-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-7-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-7-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-7-1" class="tab-pane active" aria-labelledby="tabset-7-1-tab">
<div class="cell">
<pre>predictions &lt;- predict(fit, split = &quot;validation&quot;)

files_of &lt;- function(case, what) {
  list.files(file.path(data_dir, case, what), full.names = TRUE)
}
truth_of &lt;- function(case) {
  masks &lt;- lapply(files_of(case, &quot;masks&quot;), function(f) {
    read_masks(f, binary_colormap, size = c(256, 256))[1, , ]
  })
  unite_masks(masks)
}

middle &lt;- which.min(abs(cases$dice - median(cases$dice)))
shown &lt;- c(which(cases$kind == &quot;brightfield&quot;), middle)
images &lt;- read_images(
  vapply(cases$case[shown], function(k) files_of(k, &quot;images&quot;)[1], &quot;&quot;),
  size = c(256, 256)
)
truth &lt;- simplify2array(lapply(cases$case[shown], truth_of))

plot_masks(
  images,
  prediction = predictions[shown, , ],
  truth      = aperm(truth, c(3, 1, 2)),
  colormap   = binary_colormap,
  labels     = sprintf(&quot;%s, Dice %.2f&quot;, cases$kind[shown], cases$dice[shown])
)</pre>
</div>
</div>
<div id="tabset-7-2" class="tab-pane" aria-labelledby="tabset-7-2-tab">
<pre>predictions = engine.predict(&quot;nuclei&quot;, &quot;validation&quot;)
data = engine.dataset(engine.runs[&quot;nuclei&quot;].spec, &quot;validation&quot;)

middle = np.median([r[&quot;dice&quot;] for r in cases])
shown = [i for i, r in enumerate(cases) if r[&quot;kind&quot;] == &quot;brightfield&quot;]
shown.append(min(range(len(cases)),
                 key=lambda i: abs(cases[i][&quot;dice&quot;] - middle)))

pp.plot_masks(
    np.stack([data[i][0] for i in shown]),
    prediction=predictions[shown],
    truth=np.stack([data[i][1] for i in shown]),
    labels=[f&quot;{cases[i]['kind']}, Dice {cases[i]['dice']:.2f}&quot;
            for i in shown],
)</pre>
</div>
</div>
</div>
<div class="cell">
<div class="cell-output-display">
<div id="fig-worst" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-worst-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://i2.wp.com/datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/index_files/figure-html/fig-worst-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-worst-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 2: The three brightfield images, and a fluorescence image from the middle of the ranking. In the last column green is found, red is missed and yellow is invented.
</figcaption>
</figure>
</div>
</div>
</div>
</section>
<section id="the-kind-of-microscope-or-the-size-of-a-nucleus" class="level2">
<h2 class="anchored" data-anchor-id="the-kind-of-microscope-or-the-size-of-a-nucleus">The kind of microscope, or the size of a nucleus?</h2>
<p>The pictures suggest a second reading. The brightfield images are 1024 by 1024 pixels, and the model sees every image at 256 by 256 - so a nucleus sixteen pixels across arrives as four. Measure the average nucleus on the model’s grid:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-8-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-8-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-8-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-8-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-8-1" class="tab-pane active" aria-labelledby="tabset-8-1-tab">
<div class="cell">
<pre>nucleus_area &lt;- function(case) {
  files &lt;- list.files(file.path(data_dir, case, &quot;masks&quot;), full.names = TRUE)
  areas &lt;- vapply(files, function(f) {
    sum(read_masks(f, binary_colormap) == 2)
  }, numeric(1))
  size &lt;- dim(read_masks(files[1], binary_colormap))[2:3]
  mean(areas) * 256^2 / prod(size)
}
cases$area &lt;- vapply(cases$case, nucleus_area, numeric(1))

aggregate(area ~ kind, cases, function(x) round(range(x)))
#&gt;           kind area.1 area.2
#&gt; 1  brightfield     11     17
#&gt; 2 fluorescence     69    957
#&gt; 3      stained    137   1893</pre>
</div>
</div>
<div id="tabset-8-2" class="tab-pane" aria-labelledby="tabset-8-2-tab">
<pre>def nucleus_area(case):
    files = sorted((Path(data_dir) / case / &quot;masks&quot;).glob(&quot;*.png&quot;))
    masks = [np.asarray(Image.open(f)) &gt; 0 for f in files]
    height, width = masks[0].shape[:2]
    return np.mean([m.sum() for m in masks]) * 256**2 / (height * width)

for row in cases:
    row[&quot;area&quot;] = nucleus_area(row[&quot;case&quot;])

areas = {k: [r[&quot;area&quot;] for r in cases if r[&quot;kind&quot;] == k] for k in kinds}
{k: (round(min(a)), round(max(a))) for k, a in areas.items()}
#&gt; {'brightfield': (11, 17), 'fluorescence': (69, 957),
#&gt;  'stained': (137, 1893)}</pre>
</div>
</div>
</div>
<p>On every other image an average nucleus covers at least 69 pixels of the grid. On the three brightfield images it covers eleven to seventeen. So the three images are both the only brightfield in the split and the only ones where the nuclei are this small, and <strong>three images cannot tell those two explanations apart</strong>. Training does not help either: of the 536 training images, 13 are brightfield, all of them 1024 by 1024 as well.</p>
<p>Scoring at the files’ own resolution does not separate them either, because the detail was lost before the score was taken - the prediction was made at 256:</p>
<div class="tabset-margin-container"></div><div class="panel-tabset" data-group="language">
<ul class="nav nav-tabs"><li class="nav-item"><a class="nav-link active" id="tabset-9-1-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-9-1" aria-selected="true" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">R</a></li><li class="nav-item"><a class="nav-link" id="tabset-9-2-tab" data-bs-toggle="tab" data-bs- aria-controls="tabset-9-2" aria-selected="false" href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/" rel="nofollow" target="_blank">Python</a></li></ul>
<div class="tab-content" data-group="language">
<div id="tabset-9-1" class="tab-pane active" aria-labelledby="tabset-9-1-tab">
<div class="cell">
<pre>summary(evaluate_cases(fit, at = &quot;source&quot;))
#&gt;   metric   n     mean         sd    median       min       max
#&gt; 1   dice 134 0.920808 0.05371718 0.9313503 0.7164674 0.9860661
#&gt;                                                         worst_case
#&gt; 1 7f38885521586fc6011bef1314a9fb2aa1e4935bd581b2991e1d963395eab770</pre>
</div>
</div>
<div id="tabset-9-2" class="tab-pane" aria-labelledby="tabset-9-2-tab">
<pre>pp.summarise_cases(engine.evaluate_cases(&quot;nuclei&quot;, at=&quot;source&quot;))
#&gt; [{'metric': 'dice', 'n': 134, 'mean': 0.920808045276955,
#&gt;   'sd': 0.053717183966230486, 'median': 0.9313502907752991,
#&gt;   'min': 0.7164673805236816, 'max': 0.9860661029815674,
#&gt;   'worst_case':
#&gt;     '7f38885521586fc6011bef1314a9fb2aa1e4935bd581b2991e1d963395eab770'}]</pre>
</div>
</div>
</div>
<p>What would separate them is a model that sees these images at a resolution where a nucleus is as large as it is everywhere else - tiling instead of resizing, which is a post of its own later in the series.</p>
</section>
<section id="what-to-take-from-this" class="level2">
<h2 class="anchored" data-anchor-id="what-to-take-from-this">What to take from this</h2>
<p>The mean was not wrong. 0.92 is what this model scores on these 134 images. But it is mostly an average over 105 fluorescence images, and the three images it does worst on share two properties at once, either of which could be the cause. None of that is visible in one number; all of it is visible in a table with one row per image and a column you add yourself.</p>
<p>If you report a segmentation score, report the worst case beside the mean, and look at what the worst cases have in common. Here it took one function call and a few lines of R or Python.</p>
</section>
<section id="the-series-so-far" class="level2">
<h2 class="anchored" data-anchor-id="the-series-so-far">The series so far</h2>
<ol type="1">
<li><a href="https://datascienceguts.com/posts/2026-11-03-rplatypus/" rel="nofollow" target="_blank">rplatypus rebuilt: deep learning for medical images in R</a> - why the 2020 package had to be rewritten, how it now installs its own Python, and a first mask from the same published weights used here.</li>
<li><strong>Nuclei, five years later</strong> - this post.</li>
</ol>
<p>The next one stays with segmentation but steps back from a single dataset: what rplatypus offers for it - four architectures and how they differ, the losses and metrics to choose from, and the YAML file that describes a whole experiment, field by field.</p>


</section>

 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/"> Data Science Guts</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/nuclei-five-years-later-what-a-mean-dice-hides/">Nuclei, five years later: what a mean Dice hides</a>]]></content:encoded>
					
		
		<enclosure url="https://datascienceguts.com/posts/2026-11-06-nuclei-mean-dice/thumbnail.png" length="0" type="image/png" />

		<post-id xmlns="com-wordpress:feed-additions:1">404718</post-id>	</item>
		<item>
		<title>Postal codes and tessellations</title>
		<link>https://www.r-bloggers.com/2026/11/postal-codes-and-tessellations/</link>
		
		<dc:creator><![CDATA[Diego HH]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 23:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://dieghernan.github.io/202611_postal-codes/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; "> From five imaginary points to a map of Slovenia: postal code areas with  Voronoi, giscoR and sf.</div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/postal-codes-and-tessellations/">Postal codes and tessellations</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://dieghernan.github.io/202611_postal-codes/"> One World | R packages, maps and open-source projects - R Project</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>

		
		<h3>From postal points to areas using Voronoi polygons</h3>
		
  		
			<img src='https://dieghernan.github.io/assets/img/blog/2026-11-06-postal-codes-slovenia-postal-codes-1.webp' >
			
	<p>13 min.</p>
			<p>I recently published my project <a href="https://dieghernan.github.io/codigospostales/" rel="nofollow" target="_blank">Postal codes of
Spain</a>, with estimated postal
areas, an interactive map and GeoPackage downloads. But there is something we
have not looked at yet: how do those address points become polygons? The answer
is… <strong>Voronoi polygons</strong>.</p>

<h2 id="what-is-a-voronoi-polygon">What is a Voronoi polygon?</h2>

<p>Imagine a few shops on a flat map. Which one is closest to your house? Ask that
question for every location and give each shop its own color. The result is a
<strong>Voronoi diagram</strong>, also known as a <strong>Voronoi tessellation</strong>, <strong>Voronoi
decomposition</strong>, <strong>Voronoi partition</strong> or <strong>Dirichlet tessellation</strong>. This
technique divides a plane into regions based on the distance to a given set of
points.</p>

<p>In our example, each shop is a <strong>site</strong>, or generating point. Its <strong>Voronoi
cell</strong>, also called a <strong>Thiessen polygon</strong>, contains the locations at least as
close to that site as to any other. Along a shared edge, two sites are
equally close. Where several cells meet, there may be a tie between more sites.
With ordinary straight-line distance, the edges follow perpendicular bisectors
between sites. Some cells extend indefinitely, so maps usually show them clipped
to a study area.</p>

<p>The name honors Georgy Voronoi, who studied these partitions in the early
twentieth century. The idea has earlier roots, including Dirichlet’s
nineteenth-century work (hence the name Dirichlet tessellation). <a href="https://doi.org/10.2478/pcr-2018-0009" rel="nofollow" target="_blank">Pokojski and
Pokojska (2018)</a> review the history and
geographical applications.</p>

<p>These partitions are useful for exploring proximity to shops or public services,
assigning areas to weather stations and studying spatial patterns. For example,
Thiessen polygons can supply area weights when estimating rainfall from rain
gauges. The underlying question is always about the chosen distance: the nearest
hospital in a straight line may not be the quickest one to reach.</p>

<h3 id="a-tiny-example-in-r">A tiny example in <strong>R</strong></h3>

<p>Let’s start with five points in an imaginary square, including one at its
center, using just <strong>sf</strong> and <strong>ggplot2</strong>. The coordinates have arbitrary units
and no geographical coordinate reference system.</p>

<pre>library(sf)
library(ggplot2)

# Define a common theme for the maps.
background &lt;- &quot;#faf8f2&quot;
ink &lt;- &quot;#203b47&quot;

theme_set(
  theme_void() +
    theme(
      plot.background = element_rect(fill = background, colour = NA),
      panel.background = element_rect(fill = background, colour = NA),
      plot.title = element_text(face = &quot;bold&quot;, colour = ink),
      plot.subtitle = element_text(colour = ink),
      plot.caption = element_text(colour = &quot;grey40&quot;, hjust = 0),
      legend.position = &quot;bottom&quot;,
      plot.margin = margin(10, 10, 10, 10)
    )
)

sites &lt;- data.frame(
  site = c(&quot;A&quot;, &quot;B&quot;, &quot;C&quot;, &quot;D&quot;, &quot;E&quot;),
  x = c(2, 7, 3, 8, 5),
  y = c(2, 3, 8, 7, 5)
)

toy_points &lt;- st_as_sf(sites, coords = c(&quot;x&quot;, &quot;y&quot;))

study_area &lt;- st_as_sfc(st_bbox(c(xmin = 0, ymin = 0, xmax = 10, ymax = 10)))

toy_cells &lt;- toy_points |&gt;
  st_union() |&gt;
  st_voronoi(envelope = study_area) |&gt;
  st_collection_extract(&quot;POLYGON&quot;) |&gt;
  st_intersection(study_area)

ggplot() +
  geom_sf(
    data = toy_cells,
    fill = c(&quot;#a8dadc&quot;, &quot;#f4c89a&quot;, &quot;#c6b8df&quot;, &quot;#bfd59b&quot;, &quot;#f2b5c4&quot;),
    colour = &quot;white&quot;, linewidth = 1
  ) +
  geom_sf(data = toy_points, colour = &quot;#203b47&quot;, size = 3) +
  geom_text(
    data = sites, aes(x, y, label = site),
    nudge_y = 0.5, colour = &quot;#203b47&quot;, fontface = &quot;bold&quot;
  ) +
  coord_sf(datum = NA) +
  labs(
    title = &quot;Five points, five Voronoi cells&quot;,
    subtitle = &quot;Each location belongs to the closest site&quot;
  )
</pre>

<p><img src="https://dieghernan.github.io/assets/img/blog/2026-11-06-postal-codes-voronoi-toy-example-1.webp" alt="Five labeled sites in a square: A and B below, C and D above, and E at the
center, surrounded by its own bounded Voronoi cell.
" width="450" /></p>

<p>Five points, five cells. The point E at <code>(5, 5)</code> has a bounded cell
surrounded by the other four. The outer cells extend beyond the square and are
clipped to its edges. <code>st_union()</code> collects the points into one geometry,
<code>st_voronoi()</code> builds the diagram and <code>st_intersection()</code> clips it to our
square. The colors help us tell the cells apart. We have not assigned any postal
codes yet.</p>

<p>The diagram knows nothing about roads, rivers or administrative boundaries. As
we move to postal points, we will need to make our choices about those
boundaries explicit.</p>

<h2 id="before-we-start-what-are-we-estimating">Before we start: what are we estimating?</h2>

<p>The Spanish project uses CartoCiudad address points with their postal codes. It
follows <a href="https://doi.org/10.12842/WPIVIE_0322" rel="nofollow" target="_blank">Goerlich (2022)</a>, who develops an
idea presented by <a href="https://www.nosolosig.com/articulos/asi-hice-el-mapa-de-los-codigos-postales-de-espana-con-sig-y-datos-abiertos" rel="nofollow" target="_blank">Sánchez
(2020)</a>.
The <a href="https://www.uv.es/goerlich/Ivie/CodPost.html" rel="nofollow" target="_blank">updated online version</a>
describes the method in detail.</p>

<p>For Slovenia, we will use GISCO’s postal points, retrieved with <strong>giscoR</strong>.
There is an important difference here: a representative point for a code tells
us much less than thousands of addresses with that code. Detailed addresses help
the partition follow their distribution more closely, although proximity alone
still cannot reproduce postal delivery decisions.</p>

<p>We have not established that Slovenian postal areas should follow municipal
boundaries. Imposing those boundaries is a modeling choice for this exercise,
inspired by the Spanish workflow, rather than a verified property of the
Slovenian postal system.</p>

<p class="alert alert-warning p-3 mx-2 mb-3"><i class="fa fa-exclamation-triangle"></i> This post explains the workflow. The
polygons we create are not official postal boundaries, and this exercise does
not validate the GISCO postal data.</p>

<h2 id="getting-the-data">Getting the data</h2>

<p>First, let’s get the municipal boundaries and postal points. We use the 2024
release of both datasets.</p>

<pre>library(giscoR)
library(sf)
library(dplyr)

municipalities &lt;- gisco_get_lau(
  year = 2024, country = &quot;SI&quot;, epsg = 3035
) |&gt;
  select(GISCO_ID, LAU_NAME) |&gt;
  st_make_valid()

raw_points &lt;- gisco_get_postalcodes(
  year = 2024, country = &quot;SI&quot;, epsg = 3035
)
</pre>

<p>We work in <strong>ETRS89 / LAEA Europe (EPSG:3035)</strong>, a projected coordinate system
with coordinates in meters. This makes the 200 m and 50 m margins used later
meaningful. Voronoi uses Euclidean distances in this plane, rather than road
distances. The projection does not preserve every distance exactly either.</p>

<p>Before we start drawing, what have we actually downloaded?</p>

<pre>postal_points &lt;- raw_points |&gt;
  mutate(POSTCODE = trimws(as.character(POSTCODE))) |&gt;
  filter(!is.na(POSTCODE), POSTCODE != &quot;&quot;) |&gt;
  select(POSTCODE)

code_counts &lt;- postal_points |&gt;
  st_drop_geometry() |&gt;
  count(POSTCODE, name = &quot;points_per_code&quot;)

knitr::kable(data.frame(
  Indicator = c(
    &quot;Municipalities&quot;, &quot;Downloaded points&quot;, &quot;Points with a postal code&quot;,
    &quot;Distinct postal codes&quot;, &quot;Codes represented by one point&quot;
  ),
  Value = c(
    nrow(municipalities), nrow(raw_points), nrow(postal_points),
    nrow(code_counts), sum(code_counts$points_per_code == 1)
  )
))
</pre>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Indicator</th>
      <th style="text-align: right">Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">Municipalities</td>
      <td style="text-align: right">212</td>
    </tr>
    <tr>
      <td style="text-align: left">Downloaded points</td>
      <td style="text-align: right">557</td>
    </tr>
    <tr>
      <td style="text-align: left">Points with a postal code</td>
      <td style="text-align: right">557</td>
    </tr>
    <tr>
      <td style="text-align: left">Distinct postal codes</td>
      <td style="text-align: right">557</td>
    </tr>
    <tr>
      <td style="text-align: left">Codes represented by one point</td>
      <td style="text-align: right">557</td>
    </tr>
  </tbody>
</table>

<p>These counts give us a first look at the file, but they are not a quality check.
They do not establish whether every code is present or whether the coordinates
are correct. With one point per code, we will be drawing areas around
representatives rather than estimating them from individual addresses.</p>

<p>Two points at exactly the same coordinates cannot generate separate Voronoi
cells. We sort by postal code and keep the first point at each location, as in
the original workflow. This is a reproducible tie-breaking rule, not evidence
that the retained code is correct. The counts below show how many records we
remove and how many locations have conflicting codes.</p>

<pre>postal_points &lt;- postal_points |&gt; arrange(POSTCODE)

conflicts &lt;- postal_points |&gt;
  group_by(geometry) |&gt;
  summarise(codes = n_distinct(POSTCODE), .groups = &quot;drop&quot;) |&gt;
  filter(codes &gt; 1)

duplicates &lt;- duplicated(postal_points$geometry)

knitr::kable(data.frame(
  Indicator = c(&quot;Duplicate locations removed&quot;, &quot;Locations with conflicting codes&quot;),
  Value = c(sum(duplicates), nrow(conflicts))
))
</pre>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Indicator</th>
      <th style="text-align: right">Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">Duplicate locations removed</td>
      <td style="text-align: right">79</td>
    </tr>
    <tr>
      <td style="text-align: left">Locations with conflicting codes</td>
      <td style="text-align: right">16</td>
    </tr>
  </tbody>
</table>

<pre>
postal_points &lt;- postal_points[!duplicates, ]
</pre>

<h2 id="starting-with-ljubljana">Starting with Ljubljana</h2>

<p>Let’s pick Ljubljana and keep the points inside it. We use <code>st_within()</code> so that
a point exactly on a shared boundary is not assigned to both neighboring
municipalities.</p>

<pre>city &lt;- municipalities |&gt; filter(LAU_NAME == &quot;Ljubljana&quot;)

city_points &lt;- st_filter(postal_points, city, .predicate = st_within)

base_map &lt;- ggplot() +
  geom_sf(data = city, fill = &quot;#e8e4d8&quot;, colour = ink, linewidth = 0.4)

gisco_attr &lt;- gisco_attributions()

base_map +
  geom_sf(data = city_points, colour = &quot;#bd5137&quot;, size = 2) +
  labs(
    title = &quot;Ljubljana: the starting point&quot;,
    subtitle = &quot;A municipal boundary and points with postal codes&quot;,
    caption = gisco_attr
  )
</pre>

<p><img src="https://dieghernan.github.io/assets/img/blog/2026-11-06-postal-codes-ljubljana-1.webp" alt="Outline of Ljubljana municipality with six postal point locations marked
inside it.
" width="450" /></p>

<h3 id="building-the-cells">Building the cells</h3>

<p>Each cell contains locations at least as close to its generating point as to
any other point. Cell boundaries occur where distances to two points are equal.
For this first demonstration, we keep all interior points, without the margins used in
the national workflow.</p>

<pre>cells &lt;- city_points |&gt;
  st_union() |&gt;
  st_voronoi(envelope = st_as_sfc(st_bbox(city))) |&gt;
  st_collection_extract(&quot;POLYGON&quot;) |&gt;
  st_sf(geometry = _)

base_map +
  geom_sf(data = cells, fill = NA, colour = &quot;#bd5137&quot;, linewidth = 0.5) +
  geom_sf(data = city_points, colour = ink, size = 2) +
  labs(
    title = &quot;An area of proximity for every point&quot;,
    subtitle = &quot;Cells still extend beyond the municipal boundary&quot;,
    caption = &quot;Own elaboration using data from Eurostat / GISCO, 2024&quot;
  )
</pre>

<p><img src="https://dieghernan.github.io/assets/img/blog/2026-11-06-postal-codes-voronoi-1.webp" alt="Unclipped Voronoi cells outlined around six postal points, extending beyond
the shaded Ljubljana municipal boundary.
" width="450" /></p>

<p><code>st_union()</code> gathers the points so we build one diagram. One detail to keep in
mind: <code>envelope</code> supplies a working extent, but <strong>does not clip</strong> the cells to
the municipality. We need an intersection for that.</p>

<h3 id="clipping-and-recovering-postal-codes">Clipping and recovering postal codes</h3>

<p>It would be convenient if cells came out in the same order as the input points,
but we should not rely on that. Instead, we recover each cell’s code with a
spatial relationship before clipping. Each cell must contain exactly one
generating point.</p>

<pre>point_matches &lt;- st_intersects(cells, city_points)

cells$POSTCODE &lt;- city_points$POSTCODE[unlist(point_matches)]

clipped_geometry &lt;- st_intersection(st_geometry(cells), st_geometry(city))

city_cells &lt;- st_sf(
  POSTCODE = cells$POSTCODE[attr(clipped_geometry, &quot;idx&quot;)[, 1]],
  geometry = clipped_geometry
)

city_areas &lt;- city_cells |&gt;
  group_by(POSTCODE) |&gt;
  summarise(.groups = &quot;drop&quot;) |&gt;
  st_make_valid()

city_colours &lt;- setNames(
  hcl.colors(nrow(city_areas), palette = &quot;Dark 3&quot;),
  sort(city_areas$POSTCODE)
)
city_labels &lt;- st_sf(
  POSTCODE = city_areas$POSTCODE,
  geometry = st_point_on_surface(st_geometry(city_areas))
)

ggplot(city_areas) +
  geom_sf(aes(fill = POSTCODE), colour = background, linewidth = 0.6) +
  geom_sf(
    data = city_points, shape = 21, fill = ink,
    colour = background, size = 2.4
  ) +
  geom_sf_text(
    data = city_labels,
    aes(label = POSTCODE), colour = ink, fontface = &quot;bold&quot;, size = 3.5
  ) +
  scale_fill_manual(values = city_colours, guide = &quot;none&quot;) +
  coord_sf(datum = NA) +
  labs(
    title = &quot;From points to estimated areas&quot;,
    subtitle = &quot;Cells clipped and merged by postal code&quot;,
    caption = &quot;Teaching example · Not official postal boundaries&quot;
  )
</pre>

<p><img src="https://dieghernan.github.io/assets/img/blog/2026-11-06-postal-codes-clip-cells-1.webp" alt="Six colored areas clipped to Ljubljana, labeled 1000, 1210, 1211, 1231, 1260
and 1261, with their generating postal points marked.
" width="450" /></p>

<p>We intersect the geometry columns, then use the intersection’s <code>idx</code> matrix to
recover each cell’s postal code. This keeps the attribute assignment explicit.
Label positions are also calculated from geometries alone.</p>

<p>Grouping by postal code with <code>summarise()</code> merges the geometries. With many
addresses sharing a code, their cells would dissolve into one geometry, possibly
with several separate parts.</p>

<h2 id="from-one-municipality-to-the-whole-country">From one municipality to the whole country</h2>

<p>Ljubljana is done. Now let’s repeat the operation for each component of each
municipality. Municipalities can have separate pieces: normalizing to
<code>MULTIPOLYGON</code> and then casting to <code>POLYGON</code> lets us handle them independently,
without automatically extending a code to a piece with no points.</p>

<p>The municipality ID and name apply to every component, so we mark those
attributes as spatially constant before splitting the geometries.</p>

<pre>components &lt;- municipalities |&gt;
  st_set_agr(&quot;constant&quot;) |&gt;
  st_cast(&quot;MULTIPOLYGON&quot;) |&gt;
  st_cast(&quot;POLYGON&quot;) |&gt;
  mutate(component_id = row_number())
</pre>

<h3 id="why-use-an-inward-margin">Why use an inward margin?</h3>

<p>In the Spanish project, we first select addresses within an inward margin of
<strong>200 m</strong>. If none are found, we try <strong>50 m</strong>. This reduces the influence of
points close to a boundary, where small mismatches between layers may affect
their assignment.</p>

<p>There is no universal rule behind these numbers. Goerlich describes a 20 m
margin. The 200 m and 50 m choices come from my implementation. We keep them
here to explain that workflow, although sparse postal points make it easier to
exclude useful information. Comparing other margins would be a useful next step.</p>

<pre>select_points &lt;- function(component, postal_points) {
  for (margin in c(200, 50)) {
    interior &lt;- st_buffer(st_geometry(component), -margin)

    if (all(st_is_empty(interior))) next

    selected_points &lt;- st_filter(postal_points, interior,
      .predicate = st_within
    )

    if (nrow(selected_points) &gt; 0) {
      return(selected_points)
    }
  }

  postal_points[0, ]
}
</pre>

<p>The margin only selects points: the resulting areas cover the <strong>original</strong>
component, not the smaller buffered geometry. If both attempts fail, we leave
the component unassigned. We do not borrow the nearest point from outside the
municipality.</p>

<h3 id="three-cases-one-function">Three cases, one function</h3>

<p>There are three cases to handle. With no points, we keep the component and
assign <code>NA</code>. If all selected points share a code, we assign it to the whole
component. We only need Voronoi when several codes are present: no need to split
an area that already has a single assignment.</p>

<pre>build_areas &lt;- function(component, postal_points) {
  selected_points &lt;- select_points(component, postal_points)

  codes &lt;- unique(selected_points$POSTCODE)

  if (length(codes) &lt;= 1L) {
    component$POSTCODE &lt;- if (length(codes)) codes else NA_character_
    return(component)
  }

  cells &lt;- selected_points |&gt;
    st_union() |&gt;
    st_voronoi(envelope = st_as_sfc(st_bbox(component))) |&gt;
    st_collection_extract(&quot;POLYGON&quot;) |&gt;
    st_sf(geometry = _)

  point_matches &lt;- st_intersects(cells, selected_points)

  cells$POSTCODE &lt;- selected_points$POSTCODE[unlist(point_matches)]

  clipped_geometry &lt;- st_intersection(
    st_geometry(cells),
    st_geometry(component)
  )

  result &lt;- st_sf(
    POSTCODE = cells$POSTCODE[attr(clipped_geometry, &quot;idx&quot;)[, 1]],
    geometry = clipped_geometry
  ) |&gt;
    group_by(POSTCODE) |&gt;
    summarise(.groups = &quot;drop&quot;) |&gt;
    st_make_valid()

  result$GISCO_ID &lt;- component$GISCO_ID

  result$LAU_NAME &lt;- component$LAU_NAME

  result$component_id &lt;- component$component_id

  result
}

result &lt;- lapply(seq_len(nrow(components)), function(i) {
  build_areas(components[i, ], postal_points)
}) |&gt;
  bind_rows()
</pre>

<h3 id="handling-unassigned-areas">Handling unassigned areas</h3>

<p>Can we fill any of the gaps? A component without points may belong to a
municipality whose other components have information. We recover its code only
if the municipality has <strong>one distinct recovered code</strong>. This is an imputation
from the available data, not proof that the municipality really has just one
code.</p>

<pre>unique_codes &lt;- result |&gt;
  st_drop_geometry() |&gt;
  filter(!is.na(POSTCODE)) |&gt;
  distinct(GISCO_ID, POSTCODE) |&gt;
  add_count(GISCO_ID) |&gt;
  filter(n == 1)

unassigned_before &lt;- sum(is.na(result$POSTCODE))

index &lt;- match(result$GISCO_ID, unique_codes$GISCO_ID)

recoverable &lt;- is.na(result$POSTCODE) &#038; !is.na(index)

result$POSTCODE[recoverable] &lt;- unique_codes$POSTCODE[index[recoverable]]
</pre>

<p>Before drawing the final map, let’s tidy up the geometries and keep only
surfaces. Repairs can produce collections containing residual lines or points,
which do not belong in an area layer. Dissolving by postal code also removes
municipal boundaries between areas sharing that code.</p>

<pre>keep_surfaces &lt;- function(x) {
  geometry &lt;- st_make_valid(st_geometry(x))

  geometry &lt;- lapply(geometry, function(part) {
    if (!inherits(part, c(&quot;POLYGON&quot;, &quot;MULTIPOLYGON&quot;, &quot;GEOMETRYCOLLECTION&quot;))) {
      return(st_multipolygon())
    }

    surfaces &lt;- suppressWarnings(st_collection_extract(st_sfc(part), &quot;POLYGON&quot;))

    if (length(surfaces) == 0L || all(st_is_empty(surfaces))) {
      return(st_multipolygon())
    }

    st_cast(st_union(surfaces), &quot;MULTIPOLYGON&quot;, warn = FALSE)[[1]]
  })

  st_geometry(x) &lt;- st_sfc(geometry, crs = st_crs(x))

  x[!st_is_empty(x), ]
}

unassigned &lt;- result |&gt;
  filter(is.na(POSTCODE)) |&gt;
  keep_surfaces()

postal_areas &lt;- result |&gt;
  filter(!is.na(POSTCODE)) |&gt;
  group_by(POSTCODE) |&gt;
  summarise(.groups = &quot;drop&quot;) |&gt;
  keep_surfaces() |&gt;
  st_cast(&quot;MULTIPOLYGON&quot;)
</pre>

<h2 id="the-complete-map">The complete map</h2>

<p>For the national overview, let’s color the areas by the first digit of their
code. This keeps the map readable, without claiming that we have validated a
classification of postal regions. Gray is reserved for unassigned components,
with their own legend entry.</p>

<pre>postal_colours &lt;- c(
  &quot;1&quot; = &quot;#007F7F&quot;, &quot;2&quot; = &quot;#E87500&quot;, &quot;3&quot; = &quot;#3B6FB6&quot;, &quot;4&quot; = &quot;#A64E9A&quot;,
  &quot;5&quot; = &quot;#649A32&quot;, &quot;6&quot; = &quot;#D9AF00&quot;, &quot;8&quot; = &quot;#B94350&quot;, &quot;9&quot; = &quot;#6255A4&quot;,
  &quot;Unassigned&quot; = &quot;#727A80&quot;
)

postal_groups &lt;- sort(unique(substr(postal_areas$POSTCODE, 1, 1)))

ggplot() +
  geom_sf(data = municipalities, fill = &quot;#d5d2ca&quot;, colour = NA) +
  geom_sf(
    data = postal_areas,
    aes(fill = substr(POSTCODE, 1, 1)), colour = background, linewidth = 0.15
  ) +
  geom_sf(
    data = unassigned, aes(fill = &quot;Unassigned&quot;),
    colour = background, linewidth = 0.15
  ) +
  scale_fill_manual(
    values = postal_colours, name = &quot;Postal code group&quot;,
    breaks = c(postal_groups, &quot;Unassigned&quot;),
    labels = c(paste0(postal_groups, &quot;xxx&quot;), &quot;Unassigned&quot;)
  ) +
  coord_sf(datum = NA) +
  labs(
    title = &quot;Slovenia, point by point&quot;,
    subtitle = &quot;Proximity areas clipped to municipal boundaries&quot;,
    caption = paste(&quot;Eurostat / GISCO, 2024 · Own elaboration&quot;,
      &quot;Dark grey: unassigned areas · Estimated boundaries, not official&quot;,
      sep = &quot;\n&quot;
    )
  )
</pre>

<p><img src="https://dieghernan.github.io/assets/img/blog/2026-11-06-postal-codes-slovenia-postal-codes-1.webp" alt="Map of Slovenia showing estimated postal areas for the 20 municipalities processed in this draft. Colors identify postal prefixes, dark gray marks unassigned components and light gray shows the remaining municipal background." width="450" /></p>

<h2 id="checking-the-result">Checking the result</h2>

<p>The map is ready, but let’s check the geometry before going further. Are the
polygons valid? Do assigned and unassigned areas cover the territory? Are there
any appreciable overlaps? We allow one square meter for numerical differences in
the geometry operations.</p>

<pre>territory &lt;- st_union(st_geometry(municipalities))

coverage &lt;- st_union(c(st_geometry(postal_areas), st_geometry(unassigned)))

difference_m2 &lt;- as.numeric(sum(st_area(st_sym_difference(territory, coverage))))

overlap_m2 &lt;- as.numeric(
  sum(st_area(postal_areas)) + sum(st_area(unassigned)) - sum(st_area(coverage))
)

knitr::kable(data.frame(
  Indicator = c(
    &quot;Codes with an estimated area&quot;, &quot;Components assigned by municipal recovery&quot;,
    &quot;Unassigned records&quot;, &quot;Unassigned area (km²)&quot;,
    &quot;Coverage difference (m²)&quot;, &quot;Overlap (m²)&quot;
  ),
  Value = round(c(
    nrow(postal_areas), sum(recoverable), nrow(unassigned),
    as.numeric(sum(st_area(unassigned))) / 1e6, difference_m2,
    overlap_m2
  ), 4)
))
</pre>

<table>
  <thead>
    <tr>
      <th style="text-align: left">Indicator</th>
      <th style="text-align: right">Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="text-align: left">Codes with an estimated area</td>
      <td style="text-align: right">468.0000</td>
    </tr>
    <tr>
      <td style="text-align: left">Components assigned by municipal recovery</td>
      <td style="text-align: right">4.0000</td>
    </tr>
    <tr>
      <td style="text-align: left">Unassigned records</td>
      <td style="text-align: right">6.0000</td>
    </tr>
    <tr>
      <td style="text-align: left">Unassigned area (km²)</td>
      <td style="text-align: right">150.2033</td>
    </tr>
    <tr>
      <td style="text-align: left">Coverage difference (m²)</td>
      <td style="text-align: right">0.0000</td>
    </tr>
    <tr>
      <td style="text-align: left">Overlap (m²)</td>
      <td style="text-align: right">-0.0002</td>
    </tr>
  </tbody>
</table>

<p>These checks verify the geometry, <strong>not postal accuracy</strong>. We could cover an
entire country with valid polygons and still have a poor approximation of its
delivery areas.</p>

<h2 id="back-to-spain">Back to Spain</h2>

<p>We now have the full sequence: points, proximity areas, clipping and merging by
postal code. The same steps work with much more detailed addresses, such as the
CartoCiudad data in my <a href="https://dieghernan.github.io/projects/postal-codes-spain" rel="nofollow" target="_blank">Spanish postal codes
project</a>.</p>

<p>The input data matters as much as the algorithm. With one representative point
per code, Voronoi mainly maps distances between representatives. With address
points, it can better approximate where each code’s addresses are concentrated.
It still knows nothing about delivery routes, physical barriers or official
boundaries.</p>

<p>We have not compared these results with an independent postal source for
Slovenia. Before using them for analysis, we would need to review coverage,
coordinates, the relationship with municipalities and sensitivity to the
margins. For now, we have a working example that shows how the polygons are
built, one step at a time.</p>

<h2 id="references">References</h2>

<ul>
  <li>Pokojski, W. & Pokojska, P. (2018). Voronoi diagrams – inventor, method,
applications. <em>Polish Cartographical Review</em>, <em>50</em>(3), 141–150.
<a href="https://doi.org/10.2478/pcr-2018-0009" rel="nofollow" target="_blank">https://doi.org/10.2478/pcr-2018-0009</a>.</li>
  <li>Goerlich, F. J. (2022). Elaboración de un mapa de códigos postales de España
con recursos libres. Cómo evitar pagar por disponer de información de
referencia. <em>Working Papers Ivie</em>, no. 2022-3. València: Ivie.
<a href="https://doi.org/10.12842/WPIVIE_0322" rel="nofollow" target="_blank">https://doi.org/10.12842/WPIVIE_0322</a>.
Updated online version: <a href="https://www.uv.es/goerlich/Ivie/CodPost.html" rel="nofollow" target="_blank">https://www.uv.es/goerlich/Ivie/CodPost.html</a>.</li>
  <li>Sánchez, J. I. (2020, November 13). “Así hice el mapa de los códigos
postales de España con SIG y datos abiertos.” NoSoloSIG.
<a href="https://www.nosolosig.com/articulos/asi-hice-el-mapa-de-los-codigos-postales-de-espana-con-sig-y-datos-abiertos" rel="nofollow" target="_blank">https://www.nosolosig.com/articulos/asi-hice-el-mapa-de-los-codigos-postales-de-espana-con-sig-y-datos-abiertos</a>.</li>
  <li>Hernangómez, D. (2026). Códigos postales de España [Dataset].
<a href="https://doi.org/10.5281/zenodo.23698418" rel="nofollow" target="_blank">https://doi.org/10.5281/zenodo.23698418</a>.
<a href="https://dieghernan.github.io/codigospostales/" rel="nofollow" target="_blank">https://dieghernan.github.io/codigospostales/</a>.</li>
</ul>

		
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://dieghernan.github.io/202611_postal-codes/"> One World | R packages, maps and open-source projects - R Project</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/postal-codes-and-tessellations/">Postal codes and tessellations</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404700</post-id>	</item>
		<item>
		<title>Environmental modelling in your web browser</title>
		<link>https://www.r-bloggers.com/2026/11/environmental-modelling-in-your-web-browser/</link>
		
		<dc:creator><![CDATA[Seascapemodels]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://www.seascapemodels.org/posts/2026-11-06-environmental-modelling-in-your-web-browser/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>I recall when the R shiny framework first came out. It suddenly got relatively easy for us ecological modellers and environmental statisticians to turn our R code into web apps. This improved accessibility of complex analyses to more general aud...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/environmental-modelling-in-your-web-browser/">Environmental modelling in your web browser</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://www.seascapemodels.org/posts/2026-11-06-environmental-modelling-in-your-web-browser/"> Seascapemodels</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 





<p>I recall when the R shiny framework first came out. It suddenly got relatively easy for us ecological modellers and environmental statisticians to turn our R code into web apps. This improved accessibility of complex analyses to more general audiences.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i0.wp.com/www.seascapemodels.org/posts/2026-11-06-environmental-modelling-in-your-web-browser/FANGS.png?w=578&#038;ssl=1" class="img-fluid figure-img" data-recalc-dims="1"></p>
<figcaption>FANGS, my browser based Bayesian modelling software</figcaption>
</figure>
</div>
<p>You started seeing ‘web apps’ popping up as outcomes of all sorts of projects. (Here’s one great example, <a href="https://shiny.csiro.au/BioOceanObserver/" rel="nofollow" target="_blank">Biological Ocean Observer</a> let’s you analyse timeseries of plankton data from Australia).</p>
<p>The barrier to doing these apps before was that most of the modelling community didn’t know how to code in the languages of the web. An R based framework for web apps made that possible.</p>
<p>But there was still a downside. With R shiny you still need to host the R code on a server somewhere. This requires either a free, but limited account, or paying for compute.</p>
<p>The ideal situation is to have your app written in Java Script and html. These languages run web pages. The key difference from R Shiny is the computations run in the web browser, on the viewer’s computer. This means you can host a web page cheaply or for free (e.g. as a github page). No server for computing R required.</p>
<p>Back in 2018, we paid a JS programmer to turn my R code into a web app for one of our projects (<a href="https://mangrove-carbon.wetlands.app/" rel="nofollow" target="_blank">on projecting carbon emissions from mangrove deforestation</a>). This app become very useful for people assessing potential outcomes of carbon projects.</p>
<p>Well AI coding agents have removed that barrier coding in JS.</p>
<p>I don’t think the huge potential of this change is widely appreciated in the environmental data analysis and modelling community yet. So I wanted to call it out here.</p>
<p>The tech community itself is only just starting to appreciate the impacts of this change. <a href="https://ftrain.com/aboard-the-web-is-more-than-enough" rel="nofollow" target="_blank">Here’s a nice explanation</a> of how this new power of running apps in the browser is transformative for the tech industry. (Check out <a href="https://tinylittleoffice.com/" rel="nofollow" target="_blank">tinylittleoffice</a>, a browser based office suite that saves the document as a URL you can copy and paste, or <a href="https://simonw.github.io/battledrones/" rel="nofollow" target="_blank">Battle Drones</a> a multiplayer web game that connects two devices through sound and microphones!).</p>
<p>This change is also going to be transformative for modelling and communication of science.</p>
<p>I got inspired by examples like those in the post above, and set-off on my own AI coding experiment. The result is <a href="https://www.seascapemodels.org/FANGS/" rel="nofollow" target="_blank">FANGS</a>, a browser based Bayesian computation engine.</p>
<p>Brief background on my motivation here. I’m a huge fan of Bayesian statistics. More people should be using it. But a barrier to sharing that enthusiasm is the complexity of installing many Bayesian R packages on other people’s computers.</p>
<p>So I built (vibe coded) FANGS as an educational tool. It means my students can write and run simple Bayesian models, directly in their browsers. This means that in a 3 hour lesson I can: cover an intro to Bayesian concepts, have students write models in R like syntax, run the models and interpret the results.</p>
<p>The idea is that is enough to get them motivated to then install the proper R software. The use of model syntax that emulates the serious Bayesian R packages is also a stepping stone to the more powerful R software.</p>
<p>I’ll write up the development pipeline in a later post. But I just want to say it blew me away how well this experiment with vibe coding worked. FANGS gets you the right answers and runs faster on small teaching scale problems than the equivalent R packages.</p>
<p>I think this new power will revolutionize scientific communication and education, through improving accessibility of many types of data and science.</p>
<p>What does this new wave of scientific app development what do you need?</p>
<p>On a personal level, curiosity about what is possible in web browsers. Once you know something is possible its easy to get an AI agent to make it. Once I learned about saving data in URLs from the tinylittleoffice example, it was easy to ask Claude to implement that feature in FANGS (see the <code>copy share link</code> button).</p>
<p>On a community level, translating more of the most common R packages to Java Script libraries will help. I actually haven’t fully explored what is available in JS yet. But for example, efficient JS packages for mapping exist, extending those for complex GIS operations will extend the potential of our apps.</p>
<p>Standards for validation. When you vibe code a package, you need to know it is accurate. I’ll come to how to do this when I write up the FANGS development process.</p>
<p>Finally, a bottleneck with apps has always been user experience (UX). For any scientific application a key question for me has always been: how do we make sure users don’t misinterpret the information this app provides? I’m not sure AI can solve that problem yet. It requires thoughtful design and lots of testing.</p>
<p>Let me know cool ideas you have or have seen, I’m keen to see more examples.</p>



 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.seascapemodels.org/posts/2026-11-06-environmental-modelling-in-your-web-browser/"> Seascapemodels</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/environmental-modelling-in-your-web-browser/">Environmental modelling in your web browser</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404702</post-id>	</item>
		<item>
		<title>Shrink stored models with butcher</title>
		<link>https://www.r-bloggers.com/2026/11/shrink-stored-models-with-butcher/</link>
		
		<dc:creator><![CDATA[Marc Becker]]></dc:creator>
		<pubDate>Thu, 05 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://mlr-org.com/gallery/technical/2026-11-05-butcher/</guid>

					<description><![CDATA[<p>Scope<br />
Fitted model objects in R often carry more than they need for prediction. A glm object keeps the training data, the fitted values, the residuals, and the environment of its formula. That is fine for a single model. It adds up when resample...</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/shrink-stored-models-with-butcher/">Shrink stored models with butcher</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://mlr-org.com/gallery/technical/2026-11-05-butcher/"> mlr-org</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 




<section id="scope" class="level1">
<h1>Scope</h1>
<p>Fitted model objects in R often carry more than they need for prediction. A <code>glm</code> object keeps the training data, the fitted values, the residuals, and the environment of its formula. That is fine for a single model. It adds up when <code>resample()</code> or <code>tune()</code> keep one model per resampling iteration with <code>store_models = TRUE</code>.</p>
<p>The <a href="https://cran.r-project.org/package=butcher" rel="nofollow" target="_blank">butcher</a> package from the tidymodels ecosystem removes these components while keeping the model usable for prediction. This post shows how to apply it to a single learner and how to wrap it in a callback so that every model stored by <a href="https://mlr3.mlr-org.com/" rel="nofollow" target="_blank">mlr3</a> and <a href="https://mlr3tuning.mlr-org.com/" rel="nofollow" target="_blank">mlr3tuning</a> gets shrunk right after training.</p>
<div class="cell" data-layout-align="center">
<pre>library(mlr3)
library(mlr3learners)
library(mlr3tuning)
library(mlr3misc)
library(butcher)</pre>
</div>
<p>We measure the size of an object by serializing it. This is the number of bytes the object needs on disk or when it is sent to a parallel worker.</p>
<div class="cell" data-layout-align="center">
<pre>serialized_size = function(x) {
  round(length(serialize(x, connection = NULL)) / 1024^2, digits = 2L)
}</pre>
</div>
</section>
<section id="butcher-a-single-model" class="level1">
<h1>Butcher a single model</h1>
<p>We train a logistic regression on the spam task with 4601 emails and 57 features.</p>
<div class="cell" data-layout-align="center">
<pre>task = tsk(&quot;spam&quot;)
learner = lrn(&quot;classif.log_reg&quot;, predict_type = &quot;prob&quot;)
learner$train(task)

serialized_size(learner$model)</pre>
<div class="cell-output cell-output-stdout">
<pre>[1] 4.44</pre>
</div>
</div>
<p>The model needs about 4.4 MB. The function <code>weigh()</code> lists the components of the model object sorted by size in MB.</p>
<div class="cell" data-layout-align="center">
<pre>weigh(learner$model)</pre>
<div class="cell-output cell-output-stdout">
<pre># A tibble: 101 × 2
   object              size
   &lt;chr&gt;              &lt;dbl&gt;
 1 qr.qr             2.16  
 2 residuals         0.333 
 3 fitted.values     0.333 
 4 linear.predictors 0.333 
 5 weights           0.333 
 6 prior.weights     0.333 
 7 y                 0.333 
 8 effects           0.0773
 9 family.variance   0.0440
10 family.dev.resids 0.0440
# &#x2139; 91 more rows</pre>
</div>
</div>
<p>The QR decomposition is the largest component. The residuals, fitted values, linear predictors, weights, and the response each take another 0.33 MB. Prediction needs only a fraction of this.</p>
<p>We predict on the task, butcher the model, and write it back into the learner.</p>
<div class="cell" data-layout-align="center">
<pre>prediction = learner$predict(task)
learner$model = butcher(learner$model)

serialized_size(learner$model)</pre>
<div class="cell-output cell-output-stdout">
<pre>[1] 2.34</pre>
</div>
</div>
<p>The model shrank to roughly half its size. The predictions of the butchered model are identical to the original predictions.</p>
<div class="cell" data-layout-align="center">
<pre>all.equal(prediction, learner$predict(task))</pre>
<div class="cell-output cell-output-stdout">
<pre>[1] TRUE</pre>
</div>
</div>
<p>The function <code>butcher()</code> only removes components that the <code>predict()</code> method of the model class does not use. It keeps the QR decomposition because <code>predict.glm()</code> needs it to compute standard errors.</p>
<div class="callout callout-style-default callout-warning callout-titled">
<div class="callout-header d-flex align-content-center">
<div class="callout-icon-container">
<i class="callout-icon"></i>
</div>
<div class="callout-title-container flex-fill">
Warning
</div>
</div>
<div class="callout-body-container callout-body">
<p>A butchered model has lost information. Functions such as <code>summary()</code>, <code>residuals()</code>, or <code>fitted()</code> may fail or return wrong results on it. Only butcher models that you need for prediction.</p>
</div>
</div>
</section>
<section id="resample-callback" class="level1">
<h1>Resample callback</h1>
<p>We now want to shrink every model that <code>resample()</code> stores. Callbacks run code at specific stages of the resampling loop. We create a resample callback with <a href="https://mlr3.mlr-org.com/reference/callback_resample.html" rel="nofollow" target="_blank"><code>mlr3::callback_resample()</code></a>. The stage <code>on_resample_end</code> runs after the learner has been trained and has made its predictions. The context object gives access to the learner of the current iteration.</p>
<div class="cell" data-layout-align="center">
<pre>callback = callback_resample(&quot;butcher&quot;,
  on_resample_end = function(callback, context) {
    context$learner$model = butcher::butcher(context$learner$model)
  }
)</pre>
</div>
<p>We call <code>butcher()</code> with the package prefix because the callback may run on a parallel worker on which <code>butcher</code> is not attached.</p>
<p>We run a 5-fold cross-validation twice on the same splits, once without and once with the callback.</p>
<div class="cell" data-layout-align="center">
<pre>resampling = rsmp(&quot;cv&quot;, folds = 5L)
resampling$instantiate(task)

rr = resample(task, learner, resampling, store_models = TRUE)
rr_butchered = resample(task, learner, resampling, store_models = TRUE, callbacks = callback)</pre>
</div>
<p>We compare the size of the stored models.</p>
<div class="cell" data-layout-align="center">
<pre>serialized_size(map(rr$learners, &quot;model&quot;))</pre>
<div class="cell-output cell-output-stdout">
<pre>[1] 19.19</pre>
</div>
<pre>serialized_size(map(rr_butchered$learners, &quot;model&quot;))</pre>
<div class="cell-output cell-output-stdout">
<pre>[1] 10.4</pre>
</div>
</div>
<p>The five butchered models need about 9 MB less. The callback only touches the stored models, so the performance estimate does not change.</p>
<div class="cell" data-layout-align="center">
<pre>rr$aggregate(msr(&quot;classif.auc&quot;))</pre>
<div class="cell-output cell-output-stdout">
<pre>classif.auc 
   0.971403 </pre>
</div>
<pre>rr_butchered$aggregate(msr(&quot;classif.auc&quot;))</pre>
<div class="cell-output cell-output-stdout">
<pre>classif.auc 
   0.971403 </pre>
</div>
</div>
<p>The stored learners still predict new data.</p>
<div class="cell" data-layout-align="center">
<pre>rr_butchered$learners[[1L]]$predict(task)</pre>
<div class="cell-output cell-output-stdout">
<pre>
── &lt;PredictionClassif&gt; for 4601 observations: ──────────────────────────────────────────────────────────────────────────
 row_ids   truth response  prob.spam prob.nonspam
       1    spam     spam 0.62852865 3.714714e-01
       2    spam     spam 0.99139839 8.601607e-03
       3    spam     spam 0.99999999 5.408266e-09
     ---     ---      ---        ---          ---
    4599 nonspam  nonspam 0.03097766 9.690223e-01
    4600 nonspam  nonspam 0.07923412 9.207659e-01
    4601 nonspam  nonspam 0.02782719 9.721728e-01</pre>
</div>
</div>
<p><code>benchmark()</code> accepts the same <code>callbacks</code> argument.</p>
</section>
<section id="tuning" class="level1">
<h1>Tuning</h1>
<p>Tuning with <code>store_models = TRUE</code> stores one model per resampling iteration and hyperparameter configuration. Tuning instances take tuning callbacks, which can include the resample stages. We create one with <a href="https://mlr3tuning.mlr-org.com/reference/callback_batch_tuning.html" rel="nofollow" target="_blank"><code>mlr3tuning::callback_batch_tuning()</code></a>.</p>
<div class="cell" data-layout-align="center">
<pre>callback_tuning = callback_batch_tuning(&quot;butcher&quot;,
  on_resample_end = function(callback, context) {
    context$learner$model = butcher::butcher(context$learner$model)
  }
)</pre>
</div>
<p>We tune the complexity parameter of a decision tree with 20 random configurations and a 3-fold cross-validation.</p>
<div class="cell" data-layout-align="center">
<pre>learner_rpart = lrn(&quot;classif.rpart&quot;, cp = to_tune(1e-4, 1e-1, logscale = TRUE))

instance = tune(
  tuner = tnr(&quot;random_search&quot;),
  task = task,
  learner = learner_rpart,
  resampling = rsmp(&quot;cv&quot;, folds = 3L),
  term_evals = 20L,
  store_models = TRUE,
  callbacks = callback_tuning
)</pre>
</div>
<p>The archive contains a benchmark result with the 60 butchered models.</p>
<div class="cell" data-layout-align="center">
<pre>models = map(instance$archive$benchmark_result$resample_results$resample_result, function(rr) {
  map(rr$learners, &quot;model&quot;)
})

serialized_size(models)</pre>
<div class="cell-output cell-output-stdout">
<pre>[1] 2.68</pre>
</div>
</div>
</section>
<section id="conclusion" class="level1">
<h1>Conclusion</h1>
<p>The <code>butcher</code> package removes data, calls, and environments that a model does not need for prediction. Wrapped in a callback, it shrinks every model that <code>resample()</code>, <code>benchmark()</code>, or <code>tune()</code> stores. The stored results take less memory and are faster to save to disk and to transfer from parallel workers. How much it saves depends on the model class. For model classes without a <code>butcher</code> method, <code>butcher()</code> returns the model unchanged. You can check which classes are supported in the <a href="https://butcher.tidymodels.org/articles/available-axe-methods.html" rel="nofollow" target="_blank">butcher documentation</a>.</p>


<!-- -->

</section>

 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://mlr-org.com/gallery/technical/2026-11-05-butcher/"> mlr-org</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/shrink-stored-models-with-butcher/">Shrink stored models with butcher</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404697</post-id>	</item>
		<item>
		<title>SAS to R: Why R, and Where Your Migration Budget Should Go</title>
		<link>https://www.r-bloggers.com/2026/11/sas-to-r-why-r-and-where-your-migration-budget-should-go/</link>
		
		<dc:creator><![CDATA[The Jumping Rivers Blog]]></dc:creator>
		<pubDate>Wed, 04 Nov 2026 23:59:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>This post is part of our SAS to R series:</p>
<p>Part 1: Is SAS still used, and is it worth keeping?<br />
Part 2: Why R, and where your migration budget should go (this post)<br />
Part 3: coming soon</p>
<p>If you’re moving regulated analysis from SAS to R, two ...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/sas-to-r-why-r-and-where-your-migration-budget-should-go/">SAS to R: Why R, and Where Your Migration Budget Should Go</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/"> The Jumping Rivers Blog</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>

<p>
<a href = "https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/">
<img src="https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/" width="400" style="width:400px" class="image-center" style="display: block; margin: auto;" />
</a>
</p>
<p>This post is part of our SAS to R series:</p>
<ul>
<li>Part 1: <a href="https://www.jumpingrivers.com/blog/is-sas-worth-keeping/" rel="nofollow" target="_blank">Is SAS still used, and is it worth keeping?</a></li>
<li>Part 2: Why R, and where your migration budget should go (this post)</li>
<li>Part 3: coming soon</li>
</ul>
<p>If you’re moving regulated analysis from SAS to R, two things are worth knowing up front. R is a choice you can defend: regulators don’t tell you which software to use, and R has already been through FDA review in public pilot submissions. And the language itself is the cheap part. Most of the cost and risk comes from six other places, and most of those are about planning, not programming.</p>
<p><a href="https://www.jumpingrivers.com/blog/is-sas-worth-keeping/" rel="nofollow" target="_blank">Part one</a> asked whether SAS is still worth keeping. This post covers why R, and where to put your budget and attention once you commit.</p>
<h2 id="why-r-is-a-defensible-choice">Why R is a defensible choice</h2>
<h3 id="regulators-ask-for-evidence-not-a-vendor">Regulators ask for evidence, not a vendor</h3>
<p>The FDA’s <a href="https://www.fda.gov/media/109552/download" rel="nofollow" target="_blank">Statistical Software Clarifying Statement</a> says the agency does not require any specific software for statistical analysis. It asks for the software to be documented in the submission, including version and build, and <a href="https://www.ema.europa.eu/en/ich-e9-statistical-principles-clinical-trials-scientific-guideline" rel="nofollow" target="_blank">ICH E9</a>, the international guideline on statistics in clinical trials, asks for documented testing.</p>
<p>The business implication is that the obligation stays the same whichever language you choose. What changes is who does the work. With SAS, much of the evidence arrives with the licence. With R, your organisation produces it.</p>
<h3 id="there-is-public-precedent-to-show-your-risk-committee">There is public precedent to show your risk committee</h3>
<p>The R Consortium’s <a href="https://rconsortium.github.io/submissions-wg/" rel="nofollow" target="_blank">R Submissions Working Group</a> has taken R-based tables, figures, standard analysis datasets and an interactive app through FDA review in a series of public pilots, with the code and lessons published. That is not a guarantee for every submission, but it is the most useful thing you can put in front of a risk committee.</p>
<p><img alt="Timeline of the R Consortium pilot submissions to the FDA, with each pilot’s hex sticker. Pilot 1, tables and figures: submitted November 2021, FDA response March 2022. Pilot 2, Shiny app: submitted November 2022, FDA response September 2023. Pilot 3, analysis datasets: submitted August 2023, FDA response August 2024. Pilot 4a, WebAssembly: submitted September 2024, FDA response May 2026. Pilot 4b, containers: submitted June 2025, FDA response May 2026. Pilot 5, Dataset-JSON: submitted in 2025, resubmitted January 2026." id="h-rh-i-0" src="https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/fda-pilot-timeline.svg"></p>
<p><em>Hex stickers and pilot details: <a href="https://rconsortium.github.io/submissions-wg/" rel="nofollow" target="_blank">R Consortium R Submissions Working Group</a>.</em></p>
<h3 id="the-clinical-tooling-already-exists">The clinical tooling already exists</h3>
<p>Most of what a clinical SAS team does has a maintained R equivalent, much of it built by pharma companies working together in the <a href="https://pharmaverse.org/" rel="nofollow" target="_blank">pharmaverse</a>. You are not funding the tooling, only the move.</p>
<p>One finding comes up repeatedly. A lot of “we have to keep SAS” turns out to mean “we have to produce SAS transport files”, the format regulators expect submitted data in. That format is a published specification, not a licence, and R writes it.</p>
<p><em>For your team:</em> <a href="https://cran.r-project.org/package=admiral" rel="nofollow" target="_blank"><code>admiral</code></a> and <a href="https://cran.r-project.org/package=sdtm.oak" rel="nofollow" target="_blank"><code>sdtm.oak</code></a> build ADaM and SDTM datasets, <a href="https://cran.r-project.org/package=mmrm" rel="nofollow" target="_blank"><code>mmrm</code></a> reproduces SAS mixed models, and <a href="https://cran.r-project.org/package=xportr" rel="nofollow" target="_blank"><code>xportr</code></a> and <a href="https://cran.r-project.org/package=haven" rel="nofollow" target="_blank"><code>haven</code></a> write and read v5 transport files.</p>
<aside class="advert">
<p>
Need help with R package validation to unleash the power of open source? Check out the <a href="https://www.jumpingrivers.com/litmus/?utm_source=blog&#038;utm_medium=banner&#038;utm_campaign=sas-to-r-migration-risks" rel="nofollow" target="_blank">Litmusverse suite of risk assessment tools</a>.
</p>
</aside>
<h3 id="what-about-python">What about Python?</h3>
<p>For statistical analysis and regulated reporting, R is usually the closer match to the work SAS did. Python tends to be the better answer for machine learning engineering and production services. Many organisations run both, on shared infrastructure and review processes, so choosing R for statistics does not lock in the rest of the stack.</p>
<h2 id="where-the-cost-and-risk-come-from">Where the cost and risk come from</h2>
<p>Teams often budget a SAS to R migration as a code rewrite. The rewrite is the smallest of the seven layers below. The six above it are where timelines slip and budgets get cut.</p>
<p><img alt="A stack of seven layers showing where risk comes from in a SAS to R migration. From the bottom: the R language, lowest risk; numerical defaults; data semantics; the macro layer; environment and reproducibility; validation evidence; and at the top, people and process, the highest risk." id="h-rh-i-1" src="https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/risk-stack.svg"></p>
<h3 id="1-the-two-systems-will-disagree-and-both-will-be-right">1. The two systems will disagree, and both will be right</h3>
<p>SAS and R make different default choices for the same statistical method. For example, they handle tied survival times differently. Run the same analysis in both and the numbers can differ in the last decimal places, with neither tool wrong.</p>
<p>The cost comes when nobody notices until validation, and nobody recorded which choice the analysis plan meant. The fix is cheap if you do it early: add the method choices to each analysis specification before conversion starts. That turns an open-ended rewrite into a bounded validation exercise.</p>
<p><em>For your team:</em> the cross-industry <a href="https://psiaims.github.io/CAMIS/" rel="nofollow" target="_blank">CAMIS project</a> documents these differences method by method, including <a href="https://psiaims.github.io/CAMIS/Comp/r-sas_survival.html" rel="nofollow" target="_blank">survival analysis</a> in R’s <a href="https://cran.r-project.org/package=survival" rel="nofollow" target="_blank"><code>survival</code></a> package and <a href="https://psiaims.github.io/CAMIS/Comp/r-sas_rounding.html" rel="nofollow" target="_blank">rounding</a>.</p>
<h3 id="2-data-does-not-mean-quite-the-same-thing">2. Data does not mean quite the same thing</h3>
<p>The two languages treat missing values and dates differently. A filter carried over from SAS can quietly drop rows in R, changing the denominator. The result is a table that is nearly right, which is harder to catch than one that is obviously wrong. Budget for checking the data, not just converting the code.</p>
<p><em>For your team:</em> SAS sorts missing numerics below every value and has 28 kinds of missing, while R has one <code>NA</code>. Date origins also differ (1960 against 1970), which <code>haven</code> handles on import.</p>
<h3 id="3-your-macro-library-holds-rules-nobody-wrote-down">3. Your macro library holds rules nobody wrote down</h3>
<p>As we covered in <a href="https://www.jumpingrivers.com/blog/is-sas-worth-keeping/" rel="nofollow" target="_blank">part one</a>, a mature macro library holds business rules that exist nowhere else. A line-by-line conversion reproduces the code and loses the reasoning. Recovering that intent is analysis work, and it needs people who know the SAS estate. In mature estates it can be one of the longest parts of the migration, so plan for it explicitly.</p>
<h3 id="4-reproducibility-becomes-your-responsibility">4. Reproducibility becomes your responsibility</h3>
<p>SAS gives you one vendor managing one product. R spreads that across R itself and the add-on packages (community-built libraries of functions) your analyses depend on. Being able to rerun an analysis years later is now something your team owns. The tooling is mature, and it should be mandatory from day one rather than added later.</p>
<p><em>For your team:</em> pin dependencies with <code>renv</code>, serve packages from a dated snapshot with Posit Package Manager (which we wrote about alongside <a href="https://www.jumpingrivers.com/blog/diffify-posit-package-manager/" rel="nofollow" target="_blank">diffify</a>), and containerise anything long-lived. In the UK public sector, the same idea goes by the name <a href="https://www.jumpingrivers.com/blog/reproducible-analytical-pipelines/" rel="nofollow" target="_blank">reproducible analytical pipelines</a>.</p>
<h3 id="5-proving-it-works-is-a-separate-project">5. Proving it works is a separate project</h3>
<p>Getting the right answer and being able to prove you will keep getting it are different projects, and only the first one is fun. Budget them separately.</p>
<p>The <a href="https://pharmar.org/white-paper/" rel="nofollow" target="_blank">R Validation Hub</a> set out the widely used approach: assess each package by risk rather than validating everything to the same depth. That keeps the effort proportionate. We build on the same idea in <a href="https://www.jumpingrivers.com/litmus/" rel="nofollow" target="_blank">Litmus</a>, which scores packages at function level.</p>
<p><em>For your team:</em> how the scoring works for <a href="https://www.jumpingrivers.com/blog/litmus-scoring-r-validation/" rel="nofollow" target="_blank">validation</a> and <a href="https://www.jumpingrivers.com/blog/r-validation-code-quality-litmus/" rel="nofollow" target="_blank">code quality</a>, the shorter argument in <a href="https://www.jumpingrivers.com/blog/should-i-use-your-r-pkg/" rel="nofollow" target="_blank">Should I Use Your R Package?</a>, unit testing with <a href="https://www.jumpingrivers.com/blog/r-testthat/" rel="nofollow" target="_blank">testthat</a>, and <a href="https://www.jumpingrivers.com/blog/validating-shiny-apps-in-regulated-environments/" rel="nofollow" target="_blank">validating Shiny apps in regulated environments</a>.</p>
<h3 id="6-dual-running-is-the-cost-that-gets-cut">6. Dual running is the cost that gets cut</h3>
<p>For the whole parallel period you pay for both systems, both sets of people and both sets of sign-off. This is what de-risks the migration, and it is reliably the first thing trimmed when the budget tightens. Protect it.</p>
<p>Agree in advance how close the two sets of results need to be, and why. Then check automatically rather than by eye, so a difference shows up the day it appears.</p>
<p><em>For your team:</em> <code>diffdf</code> was built for this comparison, and <code>arsenal::comparedf</code> and <code>waldo</code> cover adjacent cases. Run them in CI.</p>
<h2 id="what-to-do-before-converting-any-code">What to do before converting any code</h2>
<ol>
<li>Inventory the macro library and the business rules inside it.</li>
<li>Add method choices to every analysis specification.</li>
<li>Decide your package validation approach.</li>
<li>Budget dual running as a protected line.</li>
</ol>
<p>Each of these is manageable once it has been named and written down. That is what an assessment produces before a single line of code is converted.</p>
<p>If you are planning a migration, <a href="https://www.jumpingrivers.com/contact/" rel="nofollow" target="_blank">get in touch</a>. If your team is earlier than that, our <a href="https://www.jumpingrivers.com/training/course/why-use-r/" rel="nofollow" target="_blank">Why Use R?</a> course is written for people arriving from SAS and SPSS.</p>
<h2 id="frequently-asked-questions">Frequently asked questions</h2>
<p><strong>Is R accepted by regulators?</strong></p>
<p>The FDA does not require any specific software, provided the software used is documented with version and build and its testing is documented. The R Consortium’s pilots have taken R-based tables, figures, standard analysis datasets and an interactive app through FDA review.</p>
<p><strong>Will R give exactly the same numbers as SAS?</strong></p>
<p>Not by default, and that is normal. The two make different default choices for some methods, so results can differ slightly while both are correct. The fix is to state those choices in the analysis specification rather than hunt for a bug.</p>
<h2 id="references">References</h2>
<ul>
<li>US FDA, <a href="https://www.fda.gov/media/109552/download" rel="nofollow" target="_blank">Statistical Software Clarifying Statement</a></li>
<li>US FDA, <a href="https://www.fda.gov/media/133219/download" rel="nofollow" target="_blank">Study Data Technical Conformance Guide</a></li>
<li>R Consortium, <a href="https://rconsortium.github.io/submissions-wg/" rel="nofollow" target="_blank">R Submissions Working Group</a> and <a href="https://r-consortium.org/posts/submissions-wg-2026/" rel="nofollow" target="_blank">2026 plans and 2025 progress</a></li>
<li>PHUSE, PSI, ASA and IASCT, <a href="https://psiaims.github.io/CAMIS/" rel="nofollow" target="_blank">CAMIS: Comparing Analysis Method Implementations in Software</a></li>
<li>CAMIS, <a href="https://psiaims.github.io/CAMIS/Comp/r-sas_survival.html" rel="nofollow" target="_blank">R versus SAS: Kaplan-Meier and Cox proportional hazards</a> and <a href="https://psiaims.github.io/CAMIS/Comp/r-sas_rounding.html" rel="nofollow" target="_blank">R versus SAS rounding</a></li>
<li>R Validation Hub, <a href="https://pharmar.org/white-paper/" rel="nofollow" target="_blank">A Risk-based Approach for Assessing R Package Accuracy within a Validated Infrastructure</a></li>
<li>The R Foundation, <a href="https://www.r-project.org/doc/R-FDA.pdf" rel="nofollow" target="_blank">R: Regulatory Compliance and Validation Issues</a></li>
<li>CRAN, <a href="https://cran.r-project.org/package=mmrm" rel="nofollow" target="_blank"><code>mmrm</code></a> for mixed models for repeated measures</li>
<li>CRAN, <a href="https://cran.r-project.org/package=admiral" rel="nofollow" target="_blank"><code>admiral</code></a> and <a href="https://cran.r-project.org/package=sdtm.oak" rel="nofollow" target="_blank"><code>sdtm.oak</code></a> for ADaM and SDTM builds</li>
<li>CRAN, <a href="https://cran.r-project.org/package=xportr" rel="nofollow" target="_blank"><code>xportr</code></a> for writing v5 transport files</li>
</ul>
<p>
For updates and revisions to this article, see the <a href = "https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/">original post</a>
</p>
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.jumpingrivers.com/blog/sas-to-r-migration-risks/"> The Jumping Rivers Blog</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/sas-to-r-why-r-and-where-your-migration-budget-should-go/">SAS to R: Why R, and Where Your Migration Budget Should Go</a>]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">404658</post-id>	</item>
		<item>
		<title>Kindness, Curiosity, and Kitties: A rOpenSci 15th Anniversary Interview with Mo</title>
		<link>https://www.r-bloggers.com/2026/11/kindness-curiosity-and-kitties-a-ropensci-15th-anniversary-interview-with-mo/</link>
		
		<dc:creator><![CDATA[rOpenSci]]></dc:creator>
		<pubDate>Wed, 04 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://ropensci.org/blog/2026/11/04/celebrating-ropensci/</guid>

					<description><![CDATA[<p>As part of rOpenSci’s 15th anniversary celebrations, Community Manager Yani sits down with community member Mo to talk about how a first software review turned into years of involvement across R-Universe, the Champions Program, the Slack communi...</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/kindness-curiosity-and-kitties-a-ropensci-15th-anniversary-interview-with-mo/">Kindness, Curiosity, and Kitties: A rOpenSci 15th Anniversary Interview with Mo</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://ropensci.org/blog/2026/11/04/celebrating-ropensci/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>

<p>As part of <a href="https://ropensci.org/tags/anniversary" rel="nofollow" target="_blank">rOpenSci’s 15th anniversary</a> celebrations, Community Manager Yani sits down with community member Mo to talk about how a first software review turned into years of involvement across R-Universe, the Champions Program, the Slack community, and core package work. Mo reflects on what has kept her engaged and also offers advice for anyone just discovering this community.</p>
<ul>
<li><strong>How did you first end up in rOpenSci’s orbit, what was the very first interaction or moment that pulled you in?</strong></li>
</ul>
<p>I actually don’t recall. I think it was a natural evolution of my general career progression using R as a scientist and slowly transitioning into a research software engineer. Still working in academia, and mostly on tooling, analysis, and data management using R, rOpenSci felt like a very natural community to get involved in.</p>
<p>I believe I was first introduced to it through <a href="https://ropensci.org/author/ma%C3%ABlle-salmon/" rel="nofollow" target="_blank">Maëlle Salmon</a>, who was/is a fellow <a href="https://rladies.org/" rel="nofollow" target="_blank">RLadies+ global</a> team member. My <a href="https://github.com/ropensci/software-review/issues/522" rel="nofollow" target="_blank">first interaction may have been as a software reviewer for a package</a>, which Maëlle asked me if I was interested in doing.</p>
<ul>
<li><strong>Over the years, you’ve shown up in a lot of different ways: <a href="https://ropensci.org/blog/2023/03/30/r-universe-stars-3-en/" rel="nofollow" target="_blank">R-Universe</a>, the <a href="https://ropensci.org/champions/" rel="nofollow" target="_blank">Champions Program</a>, an active participant on the Slack, and a <a href="https://ropensci.org/blog/2026/06/01/goodpractice/" rel="nofollow" target="_blank">contributor to our core packages</a>. Looking back, how would you describe the arc of your involvement?</strong></li>
</ul>
<p>It’s been such an easy community to be involved in.
I’ve felt that every suggestion, discussion, and help request has been met with kindness, collegiality and with a true sense of people wanting to share their knowledge.
Seeing this [atmosphere] in the Slack has lowered any barrier I had for asking <em>“stupid”</em> questions, and always felt like any contribution has been genuinely valued.
I’ve also felt very humble in how some interactions have altered my own perception of myself as an Research Software Engineer (RSE).
I remember in particular when you asked me if <a href="https://ropensci.org/blog/2023/05/05/ropensci-champions-program-teams-meet-carolina-pradier-and-athanasia-monika-mowinckel/" rel="nofollow" target="_blank">I would consider being a Champion’s mentor</a>, and being so humbled. I had actually considered applying as a Champion, but didn’t have the time to set aside for it at that moment. Being asked to be a mentor instead made me shift my own understanding of my expertise level.</p>
<div class="row">
<div class="col-md-6">
<figure><img src="https://i0.wp.com/ropensci.org/blog/2026/11/04/celebrating-ropensci/positconf.jpeg?w=578&#038;ssl=1"
alt="Mo in a stage delivering a talk about R-Universe. In the screen a galaxy picture with the rOpenSci and R-Universe logo" data-recalc-dims="1"><figcaption>
<p>Mo delivering her talk about R-Universe at posit::conf</p>
</figcaption>
</figure>
</div>
<div class="col-md-6">
<figure><img src="https://i0.wp.com/ropensci.org/blog/2026/11/04/celebrating-ropensci/starofruniverse.png?w=578&#038;ssl=1"
alt="Interview with Mo about how she uses R-Universe" data-recalc-dims="1"><figcaption>
<p>Interview with Mo for the series Star of R-Universe</p>
</figcaption>
</figure>
</div>
</div>
<ul>
<li><strong>Of all the ways you’ve participated, is there one that feels like “the” rOpenSci experience for you? The one you’d point to first?</strong></li>
</ul>
<p>I actually think the Slack is the pinnacle of how it is to be part of the community.
Which is likely frustrating for anyone not in that Slack to read.
While it’s not the most active space, the activity is generally always useful and/or enjoying.
It’s like a friendly bubble of R-geeks just wanting to make cool things.</p>
<div class="col-md-6">
<figure><img src="https://i0.wp.com/ropensci.org/blog/2026/11/04/celebrating-ropensci/catslack.png?w=578&#038;ssl=1"
alt="Mo&#39;s cat pictured outside in a spot half sun/half shade with the text "Sølvi can&#39;t decide whether it&#39;s best here in the sun, or in the shade. She&#39;s swapped placed 4 times within the last 10minutes :heart_eyes_cat:&#39;"" data-recalc-dims="1"><figcaption>
<p>Mo’s message in the Slack channel <code>rcatpeople</code></p>
</figcaption>
</figure>
</div>
<ul>
<li><strong>Across all these different roles, has your sense of what rOpenSci <em>is</em>, or what it’s for, changed over time?</strong></li>
</ul>
<p>That’s a hard one.
I feel like it’s been quite stable over the years I have participated, but there have likely been subtle changes over time that are hard to put a pin in.</p>
<div class="row">
<div class="col-md-6">
<figure><img src="https://i1.wp.com/ropensci.org/blog/2026/11/04/celebrating-ropensci/authorprofile.png?w=578&#038;ssl=1"
alt="Mo&#39;s author profile on the rOpenSci website, showing a list of blog posts and interviews she hascontributed" data-recalc-dims="1"><figcaption>
<p>Mo’s author profile on the rOpenSci website</p>
</figcaption>
</figure>
</div>
<div class="col-md-6">
<figure><img src="https://i1.wp.com/ropensci.org/blog/2026/11/04/celebrating-ropensci/ggsegverse.png?w=578&#038;ssl=1"
alt="ggsegverse R-Universe, showing packages, contributors and Mo as primary author of the ggseg package" data-recalc-dims="1"><figcaption>
<p>ggsegverse R-Universe</p>
</figcaption>
</figure>
</div>
</div>
<ul>
<li><strong>Was there ever a moment you felt surprised by how much you’d ended up doing, or by where it led?</strong></li>
</ul>
<p>All the time.
It’s almost like having a second set of colleagues, that technically don’t work with me (as in at the same company), but are equally important to my work and its robustness.
Between the software review, blog posts, collaborative tutorials, Champion mentoring, rOpenSci package tooling development, discussions in Slack (let us not forget the awesome cat pictures in the cat channel), I feel like I have quite immersed myself in the community.
But I have gotten much more back that I have contributed, I believe.
Not only in the obvious ways visible, like mentor training, learning more about collaborative development etc., but also in getting feedback on my funding applications and on my personal development projects.
It’s all been invaluable for my own development.</p>
<ul>
<li><strong>How does participating in rOpenSci activities compare to other ways you’ve been involved in open source more broadly?</strong></li>
</ul>
<p>I’m not someone who has contributed a lot to open source in general.
I’m pretty much in my lane with rOpenSci, <a href="https://rladies.org/" rel="nofollow" target="_blank">RLadies+</a> and <a href="https://carpentries.org/" rel="nofollow" target="_blank">The Carpentries</a>, three organizations I feel all emulate pretty much the same core values in how to treat contributors.
This has given me the confidence to participate openly and without fear, and has also given me strength when I’ve received some less than courteous responses in other places.</p>
<p>But comparing it to certain Stack overflow and social media interactions I have had outside these communities, it’s for sure a much more wholesome and pleasant experience.
Not being afraid of being seen as incompetent for asking what someone else thinks is a basic question, is so important for how a community can keep growing and thriving.</p>
<ul>
<li><strong>What does rOpenSci mean to you at this point, fifteen years in?</strong></li>
</ul>
<p>A lot.
As I am sure you can tell from all that I write here.
rOpenSci is really my second professional “home”, where I know I can get support and feedback from those willing to take the time (and even if no one has the time, one generally always gets a positive nudge to continue).</p>
<ul>
<li><strong>What would you want someone just discovering rOpenSci today to know?</strong></li>
</ul>
<p>Don’t hesitate to contact and get involved.
Start wherever you feel you can; <a href="https://ropensci.org/software-review/" rel="nofollow" target="_blank">doing a software review as a reviewer is a nice experience</a> and a really good place to start.
You get to learn a little about reviewing others’ work, how to work with git and github, and communicate with other professionals.
It really is a very good starting point.</p>
<p><em>What a kind vision of rOpenSci! This post is part of our 15th anniversary celebration series, and we still have more to come!</em></p>
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ropensci.org/blog/2026/11/04/celebrating-ropensci/"> rOpenSci - open tools for open science</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/kindness-curiosity-and-kitties-a-ropensci-15th-anniversary-interview-with-mo/">Kindness, Curiosity, and Kitties: A rOpenSci 15th Anniversary Interview with Mo</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404671</post-id>	</item>
		<item>
		<title>Roche and Appsilon Build on Their Collaboration to Support teal’s Continued Development</title>
		<link>https://www.r-bloggers.com/2026/11/roche-and-appsilon-build-on-their-collaboration-to-support-teals-continued-development/</link>
		
		<dc:creator><![CDATA[Dony Unardi]]></dc:creator>
		<pubDate>Wed, 04 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://pharmaverse.github.io/blog/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.html</guid>

					<description><![CDATA[<p>Following the recent migration of the teal framework to pharmaverse, Roche and Appsilon are building on their existing collaboration to support its continued development. Appsilon will continue contributing developer expertise and will be recogn...</p>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/roche-and-appsilon-build-on-their-collaboration-to-support-teals-continued-development/">Roche and Appsilon Build on Their Collaboration to Support teal’s Continued Development</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://pharmaverse.github.io/blog/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.html"> pharmaverse blog</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 





<p>Following the recent migration of the teal framework to pharmaverse, Roche and Appsilon are building on their existing collaboration to support its continued development. Appsilon will continue contributing developer expertise and will be recognized as a core contributor and copyright holder alongside Roche.</p>
<p>teal helps developers build Shiny applications for exploring clinical trial data. If you’ve worked with it, you’ll know its modular approach: combining data, filters, and analysis modules into interactive applications.</p>
<p>Appsilon has been working with Roche on {teal} development and has been a core contributor to the open-source library. For example, it has co-implemented the new teal design, teal.reporter, and teal.picks, changes that have improved the quality and capabilities of the framework. Formal recognition as a core contributor is a new step forward that formalizes Appsilon’s role in the project, alongside Roche and the wider community.</p>
<section id="what-changes-for-users-and-contributors" class="level2">
<h2 class="anchored" data-anchor-id="what-changes-for-users-and-contributors">What Changes for Users and Contributors?</h2>
<p>The teal framework packages are now hosted in the pharmaverse GitHub organization, as described in the earlier migration announcement. Users of existing released versions do not need to change their R code as a result of the move. For development versions installed directly from GitHub, we recommend updating package references to their new pharmaverse locations.</p>
<p>Appsilon will continue co-leading the maintenance and development of the teal framework alongside Roche, with input and contributions from the wider community. This continued collaboration supports the framework’s ongoing development while welcoming other companies and individual contributors to help shape its direction.</p>
<p>Users and contributors can continue reporting issues, proposing features, and submitting pull requests in the package repositories. Community feedback and contributions remain an important part of teal’s development.</p>
</section>
<section id="whats-next-for-teal" class="level2">
<h2 class="anchored" data-anchor-id="whats-next-for-teal">What’s Next for teal?</h2>
<p>Future development will focus on making teal easier to contribute to and strengthening its use in validated environments. Work includes expanding guidance for AI-assisted development across the core and module packages, alongside improvements to testing and dependency management. Roche and Appsilon will continue working with the wider community to shape priorities and guide the framework’s development.</p>
</section>
<section id="building-on-the-communitys-work" class="level2">
<h2 class="anchored" data-anchor-id="building-on-the-communitys-work">Building on the Community’s Work</h2>
<p>teal was created at Roche and has grown through the work of developers and users contributing code, modules, documentation, bug reports, and feedback. Appsilon has been a vital part of the teal community and is taking on a greater role in its continued development alongside Roche and other contributors.</p>
<p>Thank you to each author of teal: Dony Unardi, Dawid Kałędkowski, Pawel Rucki, Aleksander Chlebowski, Andre Verissimo, Kartikeya Kirar, Vedha Viyash, Marcin Kosinski, Adrian Waddell, Nikolas Burkoff, Mahmoud Hallal, Maciej Nasinski, Konrad Pagacz, Junlue Zhao, Tadeusz Lewandowski, Lluis Revilla and everyone else who has helped develop and maintain the project. We look forward to continuing to work with you and welcoming new contributors.</p>
<blockquote class="blockquote">
<p>It’s been rewarding to see teal grow and support colleagues across the biotech and pharmaceutical community. Moving to pharmaverse opens a new chapter in that journey, and I’m excited to build on our longstanding collaboration with Appsilon through a shared commitment to open-source development. Together with the wider community, we look forward to shaping teal’s future and expanding what it can help users achieve.</p>
<p>— Dony Unardi, Roche (teal Product Owner and NEST Chief Engineer)</p>
</blockquote>
<blockquote class="blockquote">
<p>We are excited to welcome Appsilon as a partner for teal. Open source ecosystems like pharmaverse thrive when dedicated communities and forward-thinking organizations step up together as one &#8211; enabling powerful frameworks like teal to flourish and drive real impact. We hope this move to the pharmaverse org also encourages further contribution from across the industry, so this continues to prosper.</p>
<p>— Ross Farrugia, Roche (pharmaverse co-founder)</p>
</blockquote>
<blockquote class="blockquote">
<p>teal is a great example of the unique value of open-source software and the power of an active community. Having co-developed teal over the past two years, we are excited to begin this new chapter as copyright holders. We look forward to engaging even more closely with the community as teal moves to the Pharmaverse GitHub organization.</p>
<p>— Oriol Senan, R/Shiny developer at Appsilon, contributor to teal</p>
</blockquote>
<p>If you use teal, your feedback helps shape it. Report an issue, suggest an improvement, contribute a module, or join the conversation in the Pharmaverse Slack’s #teal channel.</p>
<p><a href="https://github.com/pharmaverse/teal" rel="nofollow" target="_blank">Explore the repository</a> · <a href="https://pharmaverse.github.io/teal/" rel="nofollow" target="_blank">Read the documentation</a> · <a href="https://github.com/pharmaverse/teal/blob/main/.github/CONTRIBUTING.md" rel="nofollow" target="_blank">Read the contribution guide</a></p>
</section>
<div class="cell">
<div class="cell-output-display">


</div>
</div>



<div id="quarto-appendix" class="default"><section id="last-updated" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Last updated</h2><div class="quarto-appendix-contents">

<p>2026-11-03 18:52:55.526917</p>
</div></section><section id="details" class="level2 appendix"><h2 class="anchored quarto-appendix-heading">Details</h2><div class="quarto-appendix-contents">

<p><a href="https://github.com/pharmaverse/blog/tree/main/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.qmd" rel="nofollow" target="_blank">Source</a>, <a href="https://pharmaverse.github.io/blog/session_info.html" rel="nofollow" target="_blank">Session info</a></p>
</div></section><section class="quarto-appendix-contents" id="quarto-reuse"><h2 class="anchored quarto-appendix-heading">Reuse</h2><div class="quarto-appendix-contents"><div><a rel="nofollow" href="https://creativecommons.org/licenses/by/4.0/" target="_blank">CC BY 4.0</a></div></div></section><section class="quarto-appendix-contents" id="quarto-citation"><h2 class="anchored quarto-appendix-heading">Citation</h2><div><div class="quarto-appendix-secondary-label">BibTeX citation:</div><pre>@online{unardi2026,
  author = {Unardi, Dony and Veríssimo, André and Senan, Oriol},
  title = {Roche and {Appsilon} {Build} on {Their} {Collaboration} to
    {Support} Teal’s {Continued} {Development}},
  date = {2026-11-04},
  url = {https://pharmaverse.github.io/blog/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.html},
  langid = {en}
}
</pre><div class="quarto-appendix-secondary-label">For attribution, please cite this work as:</div><div id="ref-unardi2026" class="csl-entry quarto-appendix-citeas">
Unardi, Dony, André Veríssimo, and Oriol Senan. 2026. <span>“Roche and
Appsilon Build on Their Collaboration to Support Teal’s Continued
Development.”</span> November 4, 2026. <a href="https://pharmaverse.github.io/blog/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.html" rel="nofollow" target="_blank">https://pharmaverse.github.io/blog/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.html</a>.
</div></div></section></div> 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://pharmaverse.github.io/blog/posts/2026-11-04-roche-appsilon-teal/roche-appsilon-teal.html"> pharmaverse blog</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/roche-and-appsilon-build-on-their-collaboration-to-support-teals-continued-development/">Roche and Appsilon Build on Their Collaboration to Support teal’s Continued Development</a>]]></content:encoded>
					
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">404632</post-id>	</item>
		<item>
		<title>L&#8217;IA travaille. Il vous reste le plus difficile.</title>
		<link>https://www.r-bloggers.com/2026/11/lia-travaille-il-vous-reste-le-plus-difficile/</link>
		
		<dc:creator><![CDATA[Boris Guarisma]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 11:49:14 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://blog.bguarisma.com/ia-travail-qui-reste</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; "> Automatiser avec l'IA ne retire pas le travail humain : il reste la vérification et le jugement, là où l'erreur de confiance est la plus probable.</div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/lia-travaille-il-vous-reste-le-plus-difficile/">L’IA travaille. Il vous reste le plus difficile.</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://blog.bguarisma.com/ia-travail-qui-reste"> Foundations</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
<p>En 1983, Lisanne Bainbridge, psychologue à l&#8217;University College London, publie un article de cinq pages devenu un classique : « <em>Les ironies de l&#8217;automatisation</em> ». Elle y observe que le concepteur qui cherche à éliminer l&#8217;opérateur lui laisse encore les tâches qu&#8217;il ne voit pas comment automatiser. Elle pensait aux procédés industriels. Sa phrase décrit tout aussi bien les équipes qui confient aujourd&#8217;hui une partie de leur travail à des agents d&#8217;IA.</p>
<p>Un dossier de projet sait chiffrer le temps que l&#8217;IA fera gagner. Il sait beaucoup moins bien chiffrer ce qui reste. Or ce qui reste n&#8217;est pas un résidu : <strong>c&#8217;est de la surveillance, de la vérification et du jugement</strong>, c&#8217;est-à-dire la part du travail où l&#8217;erreur de confiance est la plus probable. Automatiser ne retire pas le travail humain, cela en change la nature. Ce changement a un coût, mais ce coût n&#8217;est pas une fatalité : il dépend de ce qu&#8217;on retire à l&#8217;humain et de ce qu&#8217;on lui laisse.</p>
<p>Dans <a href="https://blog.bguarisma.com/prix-du-jeton-cout-tache-reussie" rel="nofollow" target="_blank">un précédent article</a>, je montrais que le <strong>coût d&#8217;une tâche réussie</strong> dépend du travail humain qui rattrape les échecs. Celui-ci pose la question suivante : <strong>que fait l&#8217;humain quand l&#8217;IA réussit ?</strong></p>
<h2>Surveiller est un travail</h2>
<p>Bainbridge en tirait une conséquence gênante : l&#8217;opérateur peut se retrouver avec « un assortiment arbitraire de tâches », celles que la machine ne sait pas faire. Il hérite aussi de la surveillance d&#8217;un système où il ne se passe presque rien.</p>
<p>On a longtemps tenu la surveillance pour une tâche peu exigeante. La recherche sur la vigilance dit le contraire. En 2008, une revue de Warm, Parasuraman et Matthews réunit des mesures comportementales, neurales et subjectives. Elles convergent : <strong>la vigilance « demande un travail mental intense et elle est stressante »</strong>. Sa charge de travail est élevée, et ces tâches réduisent l&#8217;engagement tout en augmentant la détresse. Surveiller n&#8217;est pas s&#8217;ennuyer : <strong>c&#8217;est un effort.</strong></p>
<p>Ces résultats viennent d&#8217;études expérimentales. Ils posent pourtant une question simple à tout projet d&#8217;automatisation : qui fournit cet effort, et dans quelles conditions ?</p>
<h2>Le travail glisse vers la vérification</h2>
<p><strong>L&#8217;IA générative rend cette question quotidienne</strong>. En 2025, des chercheurs de Microsoft Research et de l&#8217;université Carnegie Mellon interrogent 319 travailleurs du savoir. Selon leurs réponses, l&#8217;IA générative déplace la pensée critique « <strong>vers la vérification de l&#8217;information, l&#8217;intégration des réponses et la tenue de la tâche</strong> ». On rédige moins, on contrôle davantage. Ces résultats sont déclarés par les participants : ils disent ce que les personnes perçoivent, pas ce qu&#8217;elles font.</p>
<p>Ce qu&#8217;elles font, Xingqi Maggie Ye et Aruna Ranganathan, de Berkeley Haas, l&#8217;ont observé pendant huit mois dans une entreprise technologique de 200 personnes dont les salariés avaient largement accès aux outils d&#8217;IA générative. Leur étude ethnographique s&#8217;appuie aussi sur plus de 40 entretiens. Le temps gagné a été réinvesti dans le travail : les salariés « travaillaient plus vite, prenaient en charge un périmètre plus large et étendaient leur travail à davantage d&#8217;heures de la journée, souvent sans qu&#8217;on le leur demande ». Ye et Ranganathan notent que cette expansion semble souvent venir des salariés eux-mêmes. Une seule entreprise, des travaux encore en cours : c&#8217;est un signal, pas une loi.</p>
<p>Reste la perception. En 2025, l&#8217;organisme de recherche METR mène un essai randomisé auprès de 16 développeurs expérimentés de logiciels libres, sur 246 tâches de projets qu&#8217;ils connaissent depuis cinq ans en moyenne. Avec l&#8217;IA, ils mettent 19 % de temps en plus. Après l&#8217;étude, ils estiment pourtant que l&#8217;IA leur a fait gagner 20 % de temps. L&#8217;effectif est petit et les outils sont ceux de février à juin 2025. L&#8217;écart entre l&#8217;impression et la mesure devrait cependant alerter quiconque évalue un projet sur les seuls retours de ses utilisateurs.</p>
<h2>L&#8217;effet « bûcheron »</h2>
<p>Ce déplacement ne serait qu&#8217;une affaire d&#8217;emploi du temps s&#8217;il épargnait la qualité du jugement. <strong>Il ne l&#8217;épargne pas.</strong></p>
<p>En 2014, une méta-analyse de Linda Onnasch et de ses collègues rassemble 18 expériences. Plus le degré d&#8217;automatisation augmente, meilleure est la performance de routine ; tant que l&#8217;automate fonctionne, la charge baisse aussi, mais plus faiblement. Quand il tombe en panne, en revanche, la performance et la conscience de la situation se dégradent. Les auteurs parlent d&#8217;effet « bûcheron » : <strong>plus les arbres sont hauts, plus ils tombent loin</strong>. Dans toutes les études sauf quatre, les participants étaient des étudiants.</p>
<p><strong>L&#8217;expertise ne protège pas.</strong> En 2023, la revue <em>Radiology</em> publie une expérience menée auprès de 27 radiologues qui lisent des mammographies avec l&#8217;aide d&#8217;un système présenté comme une IA. En moyenne, les plus expérimentés classent correctement 82,3 % des clichés quand le système a raison, et 45,5 % quand il se trompe. Chez les moins expérimentés, la justesse passe de 79,7 % à 19,8 %.</p>
<p>Enfin, une revue systématique de Lyell et Coiera (2017) compare la littérature des facteurs humains et celle de la santé : sur 40 études retenues, 6 portent sur la santé. Les auteurs observent le biais d&#8217;automatisation (le fait de trop s&#8217;en remettre à l&#8217;aide à la décision) jusque dans des tâches uniques, lorsque la vérification est complexe, et l&#8217;associent à la charge cognitive de la décision. Pour le limiter, ils suggèrent de réduire cette charge.</p>
<p>Mettons ces résultats bout à bout. Quand on automatise les vérifications faciles, l&#8217;humain garde les difficiles : les exceptions, les cas ambigus, les erreurs que l&#8217;automate n&#8217;a pas vues. Ses décisions deviennent moins nombreuses, mais chacune porte sur une erreur plus rare et plus difficile à voir. C&#8217;est mon interprétation, pas un résultat mesuré : ce qui reste à vérifier après l&#8217;automatisation est ce qui expose le plus au biais.</p>
<img src="https://i1.wp.com/cdn.hashnode.com/uploads/covers/61bc41016916c3593a6b04c3/515dac84-32a0-471e-addd-2508f89472a1.png?w=578&#038;ssl=1" alt="Schéma en deux colonnes. À gauche, ce que l'automatisation retire : vérifications routinières, contrôles de forme, cas standards ; la routine devient plus fiable. À droite, ce qu'elle laisse à l'humain : jugement de pertinence, exceptions et cas ambigus, lecture des alertes, reprise en main en cas d'erreur ; les erreurs deviennent plus rares mais plus difficiles à voir." style="display:block;margin:0 auto" data-recalc-dims="1" />

<p><em>Ce que l&#8217;automatisation retire, ce qu&#8217;elle laisse. Interprétation de l&#8217;auteur à partir des études citées ; ce schéma ne représente aucune mesure.</em></p>
<h2>L&#8217;objection : l&#8217;IA allège aussi</h2>
<p>L&#8217;objection est sérieuse, et elle s&#8217;appuie sur des études solides. Erik Brynjolfsson, Danielle Li et Lindsey Raymond ont étudié l&#8217;introduction échelonnée d&#8217;un assistant conversationnel d&#8217;IA générative auprès de 5 172 agents de service client. La productivité, mesurée en demandes résolues par heure, augmente de 15 % en moyenne, <strong>avec de fortes différences</strong> : les agents les moins expérimentés et les moins qualifiés gagnent en vitesse et en qualité, les plus expérimentés et les plus qualifiés gagnent un peu en vitesse et perdent un peu en qualité. Les auteurs observent aussi une amélioration de l&#8217;expérience du travail : les clients sont plus polis et demandent moins souvent à parler à un responsable.</p>
<p>En médecine, une étude publiée en 2025 dans <em>JAMA Network Open</em> interroge 263 cliniciens de six systèmes de santé américains, avant et après trente jours d&#8217;usage d&#8217;une même plateforme de scribe IA, qui documente les consultations. La part de ceux qui déclarent un burn-out passe de 51,9 % à 38,8 %, et la charge cognitive liée aux notes diminue. L&#8217;étude n&#8217;a pas de groupe témoin, et trois de ses auteurs déclarent des liens avec l&#8217;entreprise Abridge AI, dont deux comme salariés. Elle va néanmoins dans le même sens.</p>
<p><strong>Ces résultats ne contredisent pas les précédents, ils les précisent.</strong> La méta-analyse d&#8217;Onnasch le montrait déjà : tant que l&#8217;automate fonctionne, la charge baisse. Le scribe allège une charge administrative. L&#8217;assistant propose des réponses dans une conversation. À mon sens, aucun des deux ne laisse à l&#8217;humain une vérification plus difficile que celle qu&#8217;il faisait avant. Le coût d&#8217;attention n&#8217;est donc pas une propriété de l&#8217;IA, mais de la répartition du travail : ce qu&#8217;on retire à l&#8217;humain, et ce qu&#8217;on lui laisse.</p>
<h2>Ce qu&#8217;on ne sait pas encore</h2>
<p>Trois questions restent ouvertes, et je préfère les nommer.</p>
<p>Je n&#8217;ai trouvé aucune étude qui mesure si l&#8217;effort par minute augmente quand l&#8217;humain ne garde que le jugement. L&#8217;intuition est forte, la preuve manque.</p>
<p>Sur la fatigue des personnes qui supervisent des outils d&#8217;IA au travail, la seule source directe que j&#8217;ai trouvée est une étude du BCG, menée auprès de 1 488 salariés américains de grandes entreprises et publiée dans la <em>Harvard Business Review</em>. Le BCG conseille lui-même les entreprises sur leur transformation par l&#8217;IA. L&#8217;étude conclut que <strong>l&#8217;usage excessif ou la surveillance constante des outils d&#8217;IA peut entraîner une fatigue mentale</strong>. Une seule étude, publiée par un acteur intéressé : c&#8217;est un indice, pas une preuve.</p>
<p>Enfin, je n&#8217;ai trouvé aucune étude relue par des pairs qui mesure comment la charge évolue avec le nombre d&#8217;agents qu&#8217;une même personne supervise.</p>
<h2>Trois questions avant d&#8217;automatiser</h2>
<p>On ne sait pas tout, mais on en sait assez pour poser de meilleures questions à un projet.</p>
<ol>
<li><strong>Que reste-t-il à vérifier une fois l&#8217;automatisation en place ?</strong> Pas seulement combien de temps : quelles décisions, et sont-elles plus difficiles qu&#8217;avant ?</li>
<li><strong>Qui le vérifie, et avec quel temps réellement alloué ?</strong> Une vérification sans temps alloué devient une signature, pas un contrôle.</li>
<li><strong>Que se passe-t-il le jour où l&#8217;automate se trompe ?</strong> Qui s&#8217;en aperçoit, et qui sait encore reprendre la main ?</li>
</ol>
<p>Le temps gagné reste un bon argument. Il ne dit simplement pas tout ce que l&#8217;automatisation coûte, ni à qui.</p>
<hr />
<p><em>Pour aller plus loin, le livre blanc gratuit</em> <a href="https://qognito.io/livres-blancs/la-facture-fantome-ia/" rel="nofollow" target="_blank"><em>La facture fantôme de l&#8217;IA</em></a> <em>part d&#8217;une question voisine, « qui vérifie le résultat ? », et montre comment faire apparaître la relecture, la correction et la supervision dans le coût complet d&#8217;un projet.</em></p>
<p><em>Les citations d&#8217;études publiées en anglais sont traduites par l&#8217;auteur.</em></p>
<hr />
<h2>Sources</h2>
<ul>
<li>Lisanne Bainbridge, « Ironies of automation », <em>Automatica</em>, 19(6), 1983, p. 775-779 : <a href="https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf" rel="nofollow" target="_blank">https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf</a></li>
<li>J. S. Warm, R. Parasuraman, G. Matthews, « Vigilance requires hard mental work and is stressful », <em>Human Factors</em>, 50(3), 2008, p. 433-441 : <a href="https://pubmed.ncbi.nlm.nih.gov/18689050/" rel="nofollow" target="_blank">https://pubmed.ncbi.nlm.nih.gov/18689050/</a></li>
<li>Hao-Ping (Hank) Lee et al., « The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers », CHI 2025 : <a href="https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/" rel="nofollow" target="_blank">https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/</a></li>
<li>Berkeley Haas, « AI promised to free up workers&#8217; time. UC Berkeley Haas researchers found the opposite », 18 février 2026, sur les travaux en cours de Xingqi Maggie Ye et Aruna Ranganathan, présentés dans la <em>Harvard Business Review</em> : <a href="https://newsroom.haas.berkeley.edu/ai-promised-to-free-up-workers-time-uc-berkeley-haas-researchers-found-the-opposite/" rel="nofollow" target="_blank">https://newsroom.haas.berkeley.edu/ai-promised-to-free-up-workers-time-uc-berkeley-haas-researchers-found-the-opposite/</a></li>
<li>Joel Becker et al. (METR), « Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity », 2025 : <a href="https://arxiv.org/abs/2507.09089" rel="nofollow" target="_blank">https://arxiv.org/abs/2507.09089</a></li>
<li>Linda Onnasch, Christopher D. Wickens, Huiyang Li, Dietrich Manzey, « Human Performance Consequences of Stages and Levels of Automation: An Integrated Meta-Analysis », <em>Human Factors</em>, 56(3), 2014, p. 476-488 : <a href="https://depositonce.tu-berlin.de/items/7df9879f-d77b-4f2b-8760-f0e3d174ca67/full" rel="nofollow" target="_blank">https://depositonce.tu-berlin.de/items/7df9879f-d77b-4f2b-8760-f0e3d174ca67/full</a></li>
<li>Thomas Dratsch et al., « Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance », <em>Radiology</em>, 307(4), 2023, e222176 : <a href="https://pubmed.ncbi.nlm.nih.gov/37129490/" rel="nofollow" target="_blank">https://pubmed.ncbi.nlm.nih.gov/37129490/</a></li>
<li>D. Lyell, E. Coiera, « Automation bias and verification complexity: a systematic review », <em>JAMIA</em>, 24(2), 2017, p. 423-431 : <a href="https://pubmed.ncbi.nlm.nih.gov/27516495/" rel="nofollow" target="_blank">https://pubmed.ncbi.nlm.nih.gov/27516495/</a></li>
<li>Erik Brynjolfsson, Danielle Li, Lindsey Raymond, « Generative AI at Work », <em>Quarterly Journal of Economics</em>, 140(2), 2025, p. 889-942 : <a href="https://doi.org/10.1093/qje/qjae044" rel="nofollow" target="_blank">https://doi.org/10.1093/qje/qjae044</a></li>
<li>Kristine D. Olson et al., « Use of Ambient AI Scribes to Reduce Administrative Burden and Professional Burnout », <em>JAMA Network Open</em>, 8(10), 2025, e2534976 : <a href="https://pubmed.ncbi.nlm.nih.gov/41037268/" rel="nofollow" target="_blank">https://pubmed.ncbi.nlm.nih.gov/41037268/</a></li>
<li>BCG, « When Using AI Leads to “Brain Fry” », 5 mars 2026, présentation de l&#8217;article de Julie Bedard et al. dans la <em>Harvard Business Review</em> : <a href="https://www.bcg.com/news/5march2026-when-using-ai-leads-brain-fry" rel="nofollow" target="_blank">https://www.bcg.com/news/5march2026-when-using-ai-leads-brain-fry</a></li>
</ul>

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.bguarisma.com/ia-travail-qui-reste"> Foundations</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/lia-travaille-il-vous-reste-le-plus-difficile/">L’IA travaille. Il vous reste le plus difficile.</a>]]></content:encoded>
					
		
		<enclosure url="https://cdn.hashnode.com/uploads/covers/61bc41016916c3593a6b04c3/9bfeb5f0-733c-4a7e-9c17-9e962423b7f2.jpg" length="0" type="image/jpeg" />

		<post-id xmlns="com-wordpress:feed-additions:1">404630</post-id>	</item>
		<item>
		<title>Stratified Modeling for Imbalanced Data</title>
		<link>https://www.r-bloggers.com/2026/11/stratified-modeling-for-imbalanced-data/</link>
		
		<dc:creator><![CDATA[Jason Bryer]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 05:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data.html</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>NOTE: Any updates to this article will be maintained here: https://medley.bryer.org/articles/stratified_glm.html</p>
<p>Introduction<br />
Statistical modeling when there is substantial imbalance in the dependent variable poses unique challenges. In these si...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/stratified-modeling-for-imbalanced-data/">Stratified Modeling for Imbalanced Data</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data.html"> Jason Bryer</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 




<p>NOTE: Any updates to this article will be maintained here: <a href="https://medley.bryer.org/articles/stratified_glm.html" rel="nofollow" target="_blank">https://medley.bryer.org/articles/stratified_glm.html</a></p>
<section id="introduction" class="level2">
<h2 class="anchored" data-anchor-id="introduction">Introduction</h2>
<p>Statistical modeling when there is substantial imbalance in the dependent variable poses unique challenges. In these situations models often favor the majority class. Two common approaches are upsampling and downsampling. Upsampling is the process of increasing the sampling of the minority group to more closely match the size of the majority group. Conversely, downsampling involves sampling from the majority group to more closely match the size of the minority group. The latter is generally not preferred since it artificially increases the sample size (i.e. <em>n</em>) resulting in decreasing the standard error estimates.<sup>1</sup> This paper introduces an approach where multiple logistic regression models are estimated using the minority group and a unique subset of the majority group such that all observations are majority observations are used in exactly one model while the minority group is reused across models. The final model pools the models using Rubin’s Rules defined within the context of multiple imputation (Rubin, 2004).</p>
</section>
<section id="impact-on-fitted-values" class="level2">
<h2 class="anchored" data-anchor-id="impact-on-fitted-values">Impact on Fitted Values</h2>
<p>In the United States there are broadly two types of schools, public and private. Public schools are funded by property taxes whereas privates schools charge tuition to the families. The <a href="https://www.oecd.org/en/about/programmes/pisa.html" rel="nofollow" target="_blank">Programme for International Student Assessment</a> (PISA) is an international evaluation managed by the OECD. It measures how well 15-year-old students apply their knowledge and problem-solving skills in reading, mathematics, and science to real-world challenges. PISA samples students from both public and private schools within the United States. The <code>medley</code> package contains a subset of the PISA data for the United States containing 5233 observations of 44 variables. There are 4888 public school students and 345 private school students (6.5927766%)</p>
<p>To exemplify the problem with estimating logistic regression with imbalanced data, let’s consider several models where the ratio of public-to-private schools varies from 1-to-1 to 14-to-1. Figure @ref(fig:shrinking-values) represents the distributions of the fitted values from a logistic regression with varying ratios of public-to-private schools. That is, all private school students were used in every model. For the 1-to-1 model, the first 345 public school observations, then for the 2-to-1 model the first 690 observations, etcetera until the final model uses all observations. The vertical lines correspond to the median fitted values. With all else equal, as the ratio of public-to-private schools increases, the range of fitted values shrinks towards the majority group.<sup>2</sup></p>
<div class="cell">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i0.wp.com/bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data_files/figure-html/shrinking-values-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>Distributions for varying public-to-private ratios. Vertical lines corrospond to the median of the fitted values.</figcaption>
</figure>
</div>
</div>
</div>
</section>
<section id="stratified-modeling" class="level2">
<h2 class="anchored" data-anchor-id="stratified-modeling">Stratified Modeling</h2>
<p>The goal of stratified modeling is to address imbalance in the outcome variable while still leveraging all data. The first step is to separate the majority and minority classes. Observations in the minority class will be used in all model estimations. Observations in the majority class will be randomly assigned a model. The number of models is determined by the extent of the imbalance the specified ratio of majority-to-minority observations (the default is 1). For the PISA data here, there are approximately 14 times as many public school observations to private school observations. Using a ratio of one we will estimate 14 models. It should be noted that the public schools students will be used in exactly one model whereas the private school students will be used in all models. The <code>stratified_glm()</code> performs the modeling. To maximize flexibility this function can pass any parameter to the <code>stats::glm()</code> function. However, parameters that reference specific observations (e.g. <code>weights</code>, <code>subset</code>) should not be used since <code>stats::glm()</code> will be called with subsets of the data that will not match the order of the data. Similarly, the <code>family</code> parameter defaults to <code>binomial(link='logit')</code> since this method only works with classification problems with two classes.</p>
<div class="cell">
<pre>results &lt;- medley::stratified_glm(
    formula = Private ~ .,
    data = pisa,
    ratio = 1
)</pre>
</div>
<p>The results of all the models are combined using the <code>mitools::MIcombine</code> function (Lumley, 2026). Although this approach is designed for combining multiple imputed data sets, in this case it allows us to have a single coefficient and standard error that is not artificially smaller given a larger sample size for a model. The fitted values from all the individual models is available in the <code>fitted</code> element of the returned object. Additionally, the individual model results are available in the <code>model</code> element. The <code>summary()</code> function returns a data frame with the combined coefficients, standard error, and p-value. Below only coefficients where p < 0.1 is printed to save space.</p>
<div class="cell">
<pre>pisa_summary &lt;- summary(results)
pisa_summary |&gt; dplyr::filter(p &lt; 0.1)</pre>
<div class="cell-output cell-output-stdout">
<pre>                     coefficients   se     p
(Intercept)                 -5.21 2.63 0.051
ST04Q01Male                  0.84 0.28 0.004
ST08Q04TRUE                 -0.50 0.27 0.071
ST08Q05TRUE                  0.75 0.36 0.038
ST20Q06TRUE                  1.79 0.95 0.060
ST20Q07TRUE                  0.72 0.31 0.021
ST21Q05Three or more         0.93 0.45 0.042
ST22Q01201-500 books         1.18 0.59 0.050</pre>
</div>
</div>
<p>Figure @ref(fig:fitted-distributions) plots the density distributions from the full model (single model using all data) and the stratified models. The stratified sample shows a distribution that spans the entire range of fitted values resulting in fitted values that are more interpretable.</p>
<div class="cell">
<pre>plot(results)</pre>
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data_files/figure-html/fitted-distributions-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>Distribution of fitted values for full and stratified models</figcaption>
</figure>
</div>
</div>
</div>
<p>Predictions for new data can be estimated using the <code>predict()</code> function.</p>
<div class="cell">
<pre>pisa_predicted &lt;- predict(results, newdata = pisa)
head(pisa_predicted)</pre>
<div class="cell-output cell-output-stdout">
<pre>   1    2    3    4    5    6 
0.79 0.16 0.30 0.24 0.22 0.46 </pre>
</div>
</div>
<p>Figure @ref(fig:fitted-distributions-3) shows how the stratified modeling is less effective when the ratio of public-to-private students increases to 3-to-1.</p>
<div class="cell">
<pre>medley::stratified_glm(
    formula = Private ~ .,
    data = pisa,
    ratio = 3
) |&gt; plot()</pre>
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data_files/figure-html/fitted-distributions-3-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>Distribution of fitted values when a ratio of three is used for the stratified model</figcaption>
</figure>
</div>
</div>
</div>
</section>
<section id="impact-for-predictive-modeling" class="level2">
<h2 class="anchored" data-anchor-id="impact-for-predictive-modeling">Impact for Predictive Modeling</h2>
<p>Stratified modeling clearly provides a strategy for addressing the issue of shrinking fitted values from logistic regression as imbalance in the outcome increases. In this section we will explore what impact, if any, does this approach have when modeling for prediction. To begin, we will split our data into a training and validation data set with 80% for training and 20% for validation.</p>
<div class="cell">
<pre>set.seed(2112); split &lt;- medley::stratified_split(pisa, 'Private', 0.8)
pisa_train &lt;- split$train
pisa_valid &lt;- split$validation
pisa_train$Private |&gt; table() |&gt; prop.table()</pre>
<div class="cell-output cell-output-stdout">
<pre>
FALSE  TRUE 
0.934 0.066 </pre>
</div>
<pre>pisa_valid$Private |&gt; table() |&gt; prop.table()</pre>
<div class="cell-output cell-output-stdout">
<pre>
FALSE  TRUE 
0.934 0.066 </pre>
</div>
</div>
<section id="full-model" class="level3">
<h3 class="anchored" data-anchor-id="full-model">Full Model</h3>
<p>First, we will estimate the logistic regression model using the full data set.</p>
<div class="cell">
<pre>full_model &lt;- glm(Private ~ ., data = pisa_train, family = binomial(link = 'logit'))</pre>
</div>
<p>Get the predictions for the validation data set.</p>
<div class="cell">
<pre>full_new_predictions &lt;- predict(full_model, newdata = pisa_valid, type = 'response')</pre>
</div>
<p>We calculate the metrics for a receiver operating characteristic below. The summary function provides us with some thresholds that maximize accuracy, minimize cost<sup>3</sup>, or where the specificity and sensitivity curves intersect.</p>
<div class="cell">
<pre>full_roc &lt;- medley::calculate_roc(
    predictions = full_new_predictions,
    observed = pisa_valid$Private
)
summary(full_roc)</pre>
<div class="cell-output cell-output-stdout">
<pre>AUC = 0.747
Cost of false-positive = 1
Cost of false-negative = 1
Threshold with minimum cost = 0.465 (accuracy = 93.4%)
Threshold with maximum accuracy = 0.465 (accuracy = 93.4%)
Sensitivity and specificity intersection = 0.059 (accuracy = 66.4%)</pre>
</div>
</div>
<p>Figure @ref(fig:full-roc) represents the receiver operating characteristic plot for the validation data using the full model.</p>
<div class="cell">
<pre>plot(full_roc)</pre>
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i1.wp.com/bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data_files/figure-html/full-roc-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>Receiver operating characteristic (ROC) and accuracy curves for the full model</figcaption>
</figure>
</div>
</div>
</div>
<p>In order to balance sensitivity and specificity we provide a confusion matrix using the intersection of these two curves as the cut point for determing the outcome class.</p>
<div class="cell">
<pre>medley::confusion_matrix(
    observed = pisa_valid$Private,
    predicted = full_new_predictions &gt; attr(full_roc, 'intersection')
)</pre>
<div class="cell-output cell-output-stdout">
<pre>              predicted             
  observed        FALSE         TRUE
     FALSE 649 (61.99%) 329 (31.42%)
      TRUE   23 (2.20%)   46 (4.39%)
Accuracy: 66.38%
Sensitivity: 66.36%
Specificity: 66.67%</pre>
</div>
</div>
</section>
<section id="stratified-model" class="level3">
<h3 class="anchored" data-anchor-id="stratified-model">Stratified Model</h3>
<p>Now let’s estimate the stratified model.</p>
<div class="cell">
<pre>stratified_model &lt;- medley::stratified_glm(
    formula = Private ~ .,
    data = pisa_train,
    ratio = 1
)
stratified_new_predictions &lt;- predict(stratified_model, newdata = pisa_valid)</pre>
</div>
<p>Examine the receiver operating characteristics to determine the optimal cut point. Figure @ref(fig:stratified-roc) plots the results.</p>
<div class="cell">
<pre>stratified_roc &lt;- medley::calculate_roc(
    predictions = stratified_new_predictions,
    observed = pisa_valid$Private
)
summary(stratified_roc)</pre>
<div class="cell-output cell-output-stdout">
<pre>AUC = 0.745
Cost of false-positive = 1
Cost of false-negative = 1
Threshold with minimum cost = 1 (accuracy = 93.4%)
Threshold with maximum accuracy = 1 (accuracy = 93.4%)
Sensitivity and specificity intersection = 0.509 (accuracy = 68.6%)</pre>
</div>
</div>
<div class="cell">
<pre>plot(stratified_roc)</pre>
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i2.wp.com/bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data_files/figure-html/stratified-roc-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>Receiver operating characteristic (ROC) and accuracy curves for the stratified model</figcaption>
</figure>
</div>
</div>
</div>
<p>Finally, calculate the confusion matrix using the intersection of the sensitivity and specificity curves as the cut point for determing predicted class.</p>
<div class="cell">
<pre>medley::confusion_matrix(
    observed = pisa_valid$Private,
    predicted = full_new_predictions &gt; attr(stratified_roc, 'intersection')
)</pre>
<div class="cell-output cell-output-stdout">
<pre>              predicted          
  observed        FALSE      TRUE
     FALSE 975 (93.12%) 3 (0.29%)
      TRUE   68 (6.49%) 1 (0.10%)
Accuracy: 93.22%
Sensitivity: 99.69%
Specificity: 1.45%</pre>
</div>
</div>
<p>Often introductory texts for predictive modeling and logistic regression suggest using a cut point of 0.5 for determining the class. As show above, this is not the optimal cut point when there is significant imbalance. The stratified modeling approach introduced gets us closer to that intuitive cut point in the middle. It should be noted that this approach is no better or worse for making predictions. However, it does address the issue of shrinking fitted values and makes the fitted and predicted values more interpretable.</p>
</section>
</section>
<section id="conclusion" class="level2">
<h2 class="anchored" data-anchor-id="conclusion">Conclusion</h2>
<p>The stratified modeling approach provides a modeling approach for logistic regression where interpretability of the fitted or predicted values are meaningful. For predictive modeling adjusting the cut point for making a class prediction will provide a similar result. However, this issue was initially encountered when conducting propensity score analysis (PSA). Logistic regression is often used to estimate the propensity scores, which are defined as the conditional probability of being in the treatment (Rosenbaum & Rubin, 1983). When using logistic regression, the propensity scores are simply the fitted values from the model. In this context having fitted values that span the entire range where the distribution of the treatment group (private schools in this example) is skewed to the right and the control group (public schools in this example) is skewed to the left. For these situations the stratified modeling approach may provide a reasonable approach to addressing the issue of shrinking fitted values from logistic regression with significant imbalance.</p>
</section>
<section id="references" class="level2">
<h2 class="anchored" data-anchor-id="references">References</h2>
<p>Bryer, Jason M., “A national study comparing charter and traditional public schools using propensity score analysis” (2014). <em>Legacy Theses & Dissertations (2009 - 2024).</em> 1090. https://doi.org/10.54014/H2BR-2XKN</p>
<p>Lumley, T (2026). mitools: Tools for Multiple Imputation of Missing Data_. https://doi.org/10.32614/CRAN.package.mitools. R package version 2.7, <a href="https://cran.r-project.org/package=mitools" class="uri" rel="nofollow" target="_blank">https://CRAN.R-project.org/package=mitools</a>.</p>
<p>Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. <em>Biometrika, 70</em>(1), 41–55, https://doi.org/10.1093/biomet/70.1.41</p>
<p>Rubin DB. <em>Multiple Imputation for Nonresponse in Surveys.</em> New York: John Wiley and Sons; 2004.</p>


</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>The <img src="https://latex.codecogs.com/png.latex?%5Clim_%7Bx%5Cto%5Cinfty%7D%5Cfrac%7Bn%7D%7Bx%7D=0"> for all fixed <img src="https://latex.codecogs.com/png.latex?n">.<img src="https://s.w.org/images/core/emoji/13.0.0/72x72/21a9.png" alt="↩" class="wp-smiley" style="height: 1em; max-height: 1em;" />︎</p></li>
<li id="fn2"><p>See https://psa.bryer.org/appendix-psranges.html for a simulation showing the same result.<img src="https://s.w.org/images/core/emoji/13.0.0/72x72/21a9.png" alt="↩" class="wp-smiley" style="height: 1em; max-height: 1em;" />︎</p></li>
<li id="fn3"><p>Since the costs of false-positives and false-negatives are the same this will be the same threshold as maximizing accuracy<img src="https://s.w.org/images/core/emoji/13.0.0/72x72/21a9.png" alt="↩" class="wp-smiley" style="height: 1em; max-height: 1em;" />︎</p></li>
</ol>
</section></div> 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://bryer.org/posts/2026-11-03-Stratified_Modeling_for_Imbalanced_Data.html"> Jason Bryer</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/stratified-modeling-for-imbalanced-data/">Stratified Modeling for Imbalanced Data</a>]]></content:encoded>
					
		
		<enclosure url="https://bryer.org/posts/2026-11-03-banner.png" length="0" type="image/png" />

		<post-id xmlns="com-wordpress:feed-additions:1">404655</post-id>	</item>
		<item>
		<title>Introducing tuiksdmx: A Metadata-Driven R Client for Turkish Official Statistics</title>
		<link>https://www.r-bloggers.com/2026/11/introducing-tuiksdmx-a-metadata-driven-r-client-for-turkish-official-statistics/</link>
		
		<dc:creator><![CDATA[M. Fatih Tüzen]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://mfatihtuzen.github.io/posts/2026-11-03_tuiksdmx/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>The tuiksdmx R package logo</p>
<p>1 Introduction<br />
Accessing official statistics programmatically should be straightforward. In practice, however, finding the right dataset is only part of the problem. An analyst also needs to understand its dime...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/introducing-tuiksdmx-a-metadata-driven-r-client-for-turkish-official-statistics/">Introducing tuiksdmx: A Metadata-Driven R Client for Turkish Official Statistics</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://mfatihtuzen.github.io/posts/2026-11-03_tuiksdmx/"> A Statistician&#039;s R Notebook</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 






<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://i1.wp.com/mfatihtuzen.github.io/posts/2026-11-03_tuiksdmx/logo.png?w=210&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1"></p>
<figcaption>The <code>tuiksdmx</code> R package logo</figcaption>
</figure>
</div>
<section id="introduction" class="level2" data-number="1">
<h2 data-number="1" class="anchored" data-anchor-id="introduction"><span class="header-section-number">1</span> Introduction</h2>
<p>Accessing official statistics programmatically should be straightforward. In practice, however, finding the right dataset is only part of the problem. An analyst also needs to understand its dimensions, available classifications, valid codes, and the structure of the request accepted by the data provider.</p>
<p>This is particularly relevant for <a href="https://sdmx.org/" rel="nofollow" target="_blank">SDMX</a> (Statistical Data and Metadata eXchange), the international standard widely used to exchange statistical data and metadata. SDMX provides a consistent framework for describing multidimensional statistical datasets, but it does not eliminate the need to understand their structures.</p>
<p>I developed <strong><code>tuiksdmx</code></strong>, an open-source R package that provides an <strong>authenticated, metadata-driven interface</strong> to the Turkish Statistical Institute’s (TÜİK, or TurkStat) SDMX REST service. The goal is not simply to download data. It is to make the steps leading to a valid and reproducible request easier to discover, understand, and verify.</p>
<p>The project is available on <a href="https://github.com/MFatihTuzen/tuiksdmx" rel="nofollow" target="_blank">GitHub</a>, with detailed guides and function documentation on its <a href="https://mfatihtuzen.github.io/tuiksdmx/" rel="nofollow" target="_blank">dedicated website</a>.</p>
</section>
<section id="why-a-metadata-driven-client" class="level2" data-number="2">
<h2 data-number="2" class="anchored" data-anchor-id="why-a-metadata-driven-client"><span class="header-section-number">2</span> Why a Metadata-Driven Client?</h2>
<p>An SDMX dataset is not necessarily a conventional rectangular table waiting to be downloaded. A particular <em>Dataflow</em> is associated with a <em>Data Structure Definition</em> (DSD), which specifies the dimensions used to identify a statistical series. These dimensions may reference <em>Codelists</em> defining permitted values, while <em>Concepts</em> describe what the dimensions represent.</p>
<p>A typical workflow therefore involves several questions:</p>
<ol type="1">
<li>Which Dataflow contains the statistics I need?</li>
<li>What dimensions and classifications define that Dataflow?</li>
<li>Which codes are valid for the relevant dimensions?</li>
<li>How can I request observations without assembling a raw SDMX key by hand?</li>
</ol>
<p><code>tuiksdmx</code> brings these steps together. It supports Dataflow discovery, DSD and Codelist inspection, metadata-based filter validation, structure-aware key construction, and observation retrieval. It also helps distinguish between the codes defined in metadata and those actually present in returned data.</p>
<p>The distinction matters: <strong>a code being valid does not mean that observations exist for it in every period.</strong></p>
</section>
<section id="getting-started" class="level2" data-number="3">
<h2 data-number="3" class="anchored" data-anchor-id="getting-started"><span class="header-section-number">3</span> Getting Started</h2>
<section id="install-the-package" class="level3" data-number="3.1">
<h3 data-number="3.1" class="anchored" data-anchor-id="install-the-package"><span class="header-section-number">3.1</span> Install the package</h3>
<p>Version <strong>0.1.0</strong> is distributed through GitHub; it is <strong>not currently on CRAN</strong>. Install it with <a href="https://pak.r-lib.org/" rel="nofollow" target="_blank"><code>pak</code></a>:</p>
<div class="cell">
<pre>install.packages(&quot;pak&quot;)  # If needed
pak::pak(&quot;MFatihTuzen/tuiksdmx&quot;)

library(tuiksdmx)</pre>
</div>
</section>
<section id="configure-authentication" class="level3" data-number="3.2">
<h3 data-number="3.2" class="anchored" data-anchor-id="configure-authentication"><span class="header-section-number">3.2</span> Configure authentication</h3>
<p>The TÜİK SDMX service currently requires authentication for its data endpoints. <code>tuiksdmx</code> accepts an API key and handles the authentication workflow internally. For regular use, I recommend storing the key in your personal <code>.Renviron</code> file, rather than embedding credentials in an R script.</p>
<pre>TUIK_SDMX_API_KEY=YOUR_OWN_API_KEY</pre>
<p>You can open that file using <code>usethis::edit_r_environ()</code> if you have <strong>usethis</strong> installed. Restart R after saving it. The package will then read the environment variable automatically.</p>
<p><strong>Never publish your API key or access tokens in a script, blog post, repository, or issue report.</strong> You must obtain and configure your own credentials; installing the package does not provide an API key.</p>
</section>
<section id="find-a-dataflow" class="level3" data-number="3.3">
<h3 data-number="3.3" class="anchored" data-anchor-id="find-a-dataflow"><span class="header-section-number">3.3</span> Find a Dataflow</h3>
<p>Instead of starting with a hard-coded endpoint, search the available Dataflows by subject:</p>
<div class="cell">
<pre># Search by an English description
flows &lt;- tuik_search_dataflows(
  &quot;consumer price&quot;,
  language = &quot;en&quot;,
  limit = 10
)

flows</pre>
</div>
<p>You can also search in Turkish, for example with <code>tuik_search_dataflows(&quot;tüketici fiyat&quot;)</code>. This makes discovery possible without knowing the Dataflow identifier in advance.</p>
</section>
<section id="explore-dimensions-before-retrieving-data" class="level3" data-number="3.4">
<h3 data-number="3.4" class="anchored" data-anchor-id="explore-dimensions-before-retrieving-data"><span class="header-section-number">3.4</span> Explore dimensions before retrieving data</h3>
<p>The following example uses the consumer price Dataflow <code>DF_TUFE_SDMX_TT01</code>, inspected during package development. Since provider catalogues can evolve, always check the current metadata before building a production workflow.</p>
<div class="cell">
<pre># Explore the structure without retrieving observations
tuik_explore(&quot;DF_TUFE_SDMX_TT01&quot;)

# Inspect the codes associated with one dimension
tuik_explore(
  &quot;DF_TUFE_SDMX_TT01&quot;,
  dimension = &quot;DEGISIM&quot;
)</pre>
</div>
<p>This is one of the central design decisions behind the package: <strong>understand the data structure first, then request observations.</strong> It reduces the need to guess dimension identifiers or construct query keys manually.</p>
</section>
<section id="retrieve-observations" class="level3" data-number="3.5">
<h3 data-number="3.5" class="anchored" data-anchor-id="retrieve-observations"><span class="header-section-number">3.5</span> Retrieve observations</h3>
<p>Once the relevant Dataflow and period are known, retrieving data can be concise:</p>
<div class="cell">
<pre>cpi &lt;- tuik_get_data(
  &quot;DF_TUFE_SDMX_TT01&quot;,
  start_period = &quot;2026-01&quot;,
  end_period   = &quot;2026-01&quot;
)

head(cpi)</pre>
</div>
<p>There is an important qualification: <strong>this request retrieves all series for the selected observation period</strong>, not just one CPI indicator. Restricting the time range alone does not guarantee a small response.</p>
<p>For a more focused query, the package accepts named, metadata-validated dimension filters:</p>
<div class="cell">
<pre>selected &lt;- list(
  REF_AREA = &quot;TR&quot;,
  FREQ = &quot;M&quot;,
  DEGISIM = &quot;1&quot;,
  SINIFLAMA_DUZEYI = &quot;2&quot;
)

# Validate the chosen values against the Dataflow structure
tuik_explore(
  &quot;DF_TUFE_SDMX_TT01&quot;,
  filters = selected
)</pre>
</div>
<p>A practical limitation is worth highlighting. <strong>Complete series-key filters</strong> allow the package to construct a targeted server-side SDMX request. <strong>Partial filters</strong>, on the other hand, require an explicit <code>allow_all = TRUE</code> opt-in and may lead to a broader download followed by filtering in R. The package does not present this as an efficient server-side partial-key request, nor does <code>allow_all</code> impose a data-size limit. The <a href="https://mfatihtuzen.github.io/tuiksdmx/articles/filtering-and-performance.html" rel="nofollow" target="_blank">filtering and performance guide</a> explains the distinction in detail.</p>
</section>
</section>
<section id="a-subtle-but-important-question-which-codes-actually-have-data" class="level2" data-number="4">
<h2 data-number="4" class="anchored" data-anchor-id="a-subtle-but-important-question-which-codes-actually-have-data"><span class="header-section-number">4</span> A Subtle but Important Question: Which Codes Actually Have Data?</h2>
<p>Metadata and observed data answer different questions.</p>
<p>A <strong>Codelist</strong> defines the theoretical set of codes for a dimension. But some of those codes may not appear in the observations returned for a particular period and set of filters.</p>
<p><code>tuiksdmx</code> addresses this distinction through two complementary operations:</p>
<div class="cell">
<pre>profile &lt;- tuik_flow_profile(&quot;DF_TUFE_SDMX_TT01&quot;)

# Permitted codes in the metadata
tuik_dimension_codes(profile, dimension = &quot;DEGISIM&quot;)

# Codes observed in the data for a bounded request
observed &lt;- tuik_available_codes(
  profile,
  dimension = &quot;COICOP_2018&quot;,
  start_period = &quot;2026-01&quot;,
  end_period = &quot;2026-01&quot;,
  allow_all = TRUE
)

observed</pre>
</div>
<p>The second operation inspects <strong>retrieved observations</strong>, so the result is conditional on the request. It is <strong>not</strong> an implementation of SDMX <em>ContentConstraint</em> queries. Moreover, the example deliberately permits a potentially broad download and should be used with attention to response size.</p>
<p>For statistical work, this separation helps prevent a common misunderstanding: the universe of valid categories is not necessarily the same as the categories populated by the data.</p>
</section>
<section id="documentation-and-current-scope" class="level2" data-number="5">
<h2 data-number="5" class="anchored" data-anchor-id="documentation-and-current-scope"><span class="header-section-number">5</span> Documentation and Current Scope</h2>
<p>The package includes documentation on authentication, Dataflow discovery, DSD metadata, filtering, observed availability, and code-label decoding. Rather than repeating those guides here, I recommend starting with the <a href="https://mfatihtuzen.github.io/tuiksdmx/articles/getting-started.html" rel="nofollow" target="_blank">Getting Started guide</a> and the <a href="https://mfatihtuzen.github.io/tuiksdmx/reference/" rel="nofollow" target="_blank">function reference</a>.</p>
<p>The initial version is deliberately focused. <code>tuiksdmx</code> is an SDMX client, not a general-purpose interface to every resource on the TÜİK portal. It does not claim to support every SDMX service feature or every Dataflow without further testing. Provider structures and authentication behaviour can change, so reproducible work still requires validation and sensible request boundaries.</p>
</section>
<section id="final-thoughts" class="level2" data-number="6">
<h2 data-number="6" class="anchored" data-anchor-id="final-thoughts"><span class="header-section-number">6</span> Final Thoughts</h2>
<p>When working with an API, it is tempting to judge a client only by how quickly it returns a table. For structured statistical data, I think a more useful measure is whether we can <strong>identify what we requested, validate the choices we made, and reproduce the result later</strong>.</p>
<p>That is the idea behind <code>tuiksdmx</code>: bring discovery, statistical metadata, validation, and data retrieval into the same R workflow while keeping the raw SDMX machinery largely out of the way.</p>
<p>I welcome feedback, reproducible bug reports, and contributions through the project repository. The package is an <strong>independently developed open-source project</strong>, not an official TÜİK software product.</p>
<p><strong>Project links:</strong> <a href="https://github.com/MFatihTuzen/tuiksdmx" rel="nofollow" target="_blank">GitHub repository</a> · <a href="https://mfatihtuzen.github.io/tuiksdmx/" rel="nofollow" target="_blank">Package website</a> · <a href="https://mfatihtuzen.github.io/tuiksdmx/articles/" rel="nofollow" target="_blank">Guides</a> · <a href="https://github.com/MFatihTuzen/tuiksdmx/issues" rel="nofollow" target="_blank">Issues and suggestions</a></p>
</section>
<section id="references" class="level2" data-number="7">
<h2 data-number="7" class="anchored" data-anchor-id="references"><span class="header-section-number">7</span> References</h2>
<ul>
<li>SDMX. <em>Statistical Data and Metadata eXchange</em>. <a href="https://sdmx.org/" class="uri" rel="nofollow" target="_blank">https://sdmx.org/</a>.</li>
<li>Turkish Statistical Institute (TÜİK). <em>Official website</em>. <a href="https://www.tuik.gov.tr/" class="uri" rel="nofollow" target="_blank">https://www.tuik.gov.tr/</a>.</li>
<li>Tüzen, M. F. (2026). <em>tuiksdmx: Access the Turkish Statistical Institute SDMX REST API</em> (version 0.1.0). GitHub. <a href="https://github.com/MFatihTuzen/tuiksdmx" class="uri" rel="nofollow" target="_blank">https://github.com/MFatihTuzen/tuiksdmx</a>.</li>
<li>Tüzen, M. F. <em>tuiksdmx package documentation</em>. <a href="https://mfatihtuzen.github.io/tuiksdmx/" class="uri" rel="nofollow" target="_blank">https://mfatihtuzen.github.io/tuiksdmx/</a>.</li>
</ul>


<!-- -->

</section>

 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://mfatihtuzen.github.io/posts/2026-11-03_tuiksdmx/"> A Statistician&#039;s R Notebook</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/introducing-tuiksdmx-a-metadata-driven-r-client-for-turkish-official-statistics/">Introducing tuiksdmx: A Metadata-Driven R Client for Turkish Official Statistics</a>]]></content:encoded>
					
		
		<enclosure url="https://mfatihtuzen.github.io/posts/2026-11-03_tuiksdmx/logo.png" length="0" type="image/png" />

		<post-id xmlns="com-wordpress:feed-additions:1">404653</post-id>	</item>
		<item>
		<title>rplatypus rebuilt: deep learning for medical images in R</title>
		<link>https://www.r-bloggers.com/2026/11/rplatypus-rebuilt-deep-learning-for-medical-images-in-r/</link>
		
		<dc:creator><![CDATA[Michał Maj]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 00:00:00 +0000</pubDate>
				<category><![CDATA[R bloggers]]></category>
		<guid isPermaLink="false">https://datascienceguts.com/posts/2026-11-03-rplatypus/</guid>

					<description><![CDATA[<div style = "width:60%; display: inline-block; float:left; ">
<p>In October 2020 I wrote about platypus, an R package for object detection and image segmentation: YOLOv3 and U-Net in a few lines of R, on top of the keras and tensorflow packages. People used it. It collected 134 stars and a queue of issues - ...</p></div>
<div style = "width: 40%; display: inline-block; float:right;"></div>
<div style="clear: both;"></div>
<strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/rplatypus-rebuilt-deep-learning-for-medical-images-in-r/">rplatypus rebuilt: deep learning for medical images in R</a>]]></description>
										<content:encoded><![CDATA[<!-- 
<div style="min-height: 30px;">
[social4i size="small" align="align-left"]
</div>
-->

<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 12px;">
[This article was first published on  <strong><a href="https://datascienceguts.com/posts/2026-11-03-rplatypus/"> Data Science Guts</a></strong>, and kindly contributed to <a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers</a>].  (You can report issue about the content on this page <a href="https://www.r-bloggers.com/contact-us/">here</a>)
<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div>
 






<p>In October 2020 I wrote about <a href="https://github.com/maju116/rplatypus/tree/master" rel="nofollow" target="_blank">platypus</a>, an R package for object detection and image segmentation: YOLOv3 and U-Net in a few lines of R, on top of the <code>keras</code> and <code>tensorflow</code> packages. People used it. It collected 134 stars and a queue of issues &#8211; and then it quietly stopped working, and I stopped answering them.</p>
<p>This year I went back to it. Computer vision is the part of data science I enjoy most, and this package has always been my passion project &#8211; the place where I get to learn things my day job does not ask of me. What came out is not an update but a different package, with a new name, a different backend and a narrower purpose: <strong>rplatypus</strong>, deep learning for medical images in R. It has been submitted to CRAN and is waiting for review; until it is accepted, install it from GitHub:</p>
<div class="cell">
<pre># install.packages(&quot;remotes&quot;)
remotes::install_github(&quot;maju116/rplatypus&quot;)</pre>
</div>
<p>This post is about why it had to be rewritten rather than fixed, what is in it now, and where the series of posts that follows is going.</p>
<section id="why-the-old-one-broke" class="level2">
<h2 class="anchored" data-anchor-id="why-the-old-one-broke">Why the old one broke</h2>
<p>R’s <code>keras</code> and <code>tensorflow</code> packages are not bindings to a C++ library. They are bridges, through <code>reticulate</code>, into a Python installation of TensorFlow &#8211; and in 2020 that was <em>the</em> Python installation, the one shared with everything else on the machine. Upgrade numpy for an unrelated project and an R session using platypus could stop loading. The failure never appeared in my package’s code, which made it miserable to support: the bug reports were about other people’s Python.</p>
<p>The second problem is the one that actually made me stop. platypus sat on two layers I did not control: the R <code>keras</code> and <code>tensorflow</code> packages on top of TensorFlow itself. When TensorFlow changed in Python, the R packages had to catch up before platypus could, and that often took a very long time. Until they did, my package was stuck, with issues I could not fix from my side. After enough of that, I lost the will to keep working on it.</p>
<p>Two things have changed since, and together they remove both problems at the root rather than patching around them.</p>
<p><strong>reticulate learned to isolate.</strong> Since version 1.41, a package declares its Python requirements with <code>py_require()</code>, and reticulate builds a private environment for them with <a href="https://docs.astral.sh/uv/" rel="nofollow" target="_blank">uv</a>, cached and shared with nobody. There is no common Python left for two packages to fight over. rplatypus never touches a Python anyone else uses &#8211; and the person installing it never chooses an interpreter, activates anything or matches a version. The first call that needs the engine builds the environment; every call after that reuses it. And there is no R wrapper in between any more: rplatypus calls its own engine, which I release myself, so nothing waits for anybody else to catch up.</p>
<p><strong>The medical imaging world settled on PyTorch.</strong> MONAI, nnU-Net, TorchIO, timm, the pretrained encoders and the foundation models &#8211; all of it is PyTorch. TensorFlow for medical images is effectively over, and the TensorFlow 2.9 the old engine was pinned to does not even install on Python 3.12. Staying would have meant rewriting anyway, onto a backend with no future in the field.</p>
<p>A third thing changed too. It does not fix the old failure, but it is the reason the rewrite happened at all instead of staying a good idea: <strong>writing a package got much cheaper.</strong> In 2020 I wrote every line of platypus myself, and the parts that make a package trustworthy &#8211; tests, documentation, error messages that say what went wrong &#8211; were the parts I never had time left for. This time I worked with an AI coding agent, <a href="https://claude.com/claude-code" rel="nofollow" target="_blank">Claude Code</a>.</p>
<p>I want to be precise about what that means, because “AI wrote my package” is not it. The decisions were mine: what the package is for, which datasets to measure it on, what it should refuse, and what a result has to show before it goes into the documentation &#8211; every number in this series was measured, not generated. What the agent changed was the cost of everything around those decisions: writing the tests, keeping two packages in two languages in step, chasing a failing check through reticulate, uv and CUDA. That is exactly the work that used to end side projects like this one. The rewrite took around a month.</p>
<p>So the computation moved to <a href="https://pypi.org/project/pyplatypus/" rel="nofollow" target="_blank">pyplatypus</a>, a Python engine on PyTorch, and rplatypus is the R package on top of it. The split is deliberate:</p>
<ul>
<li><strong>The engine is in Python</strong>, because that is where the models, the weights and the medical libraries already are. One codebase to maintain, and everything PyTorch has is within reach.</li>
<li><strong>The product is in R</strong>, because that is where the gap is. Reading and combining masks, plotting, comparing models, the analysis around the network &#8211; that part is plain R and needs no Python at all.</li>
</ul>
<p>I did consider native R <code>torch</code>, which binds libtorch directly and needs no Python. For one person maintaining this in their evenings it would have meant writing every architecture, loss and data pipeline twice, and giving up every pretrained weight in the Python ecosystem.</p>
</section>
<section id="pyplatypus-the-same-engine-from-python" class="level2">
<h2 class="anchored" data-anchor-id="pyplatypus-the-same-engine-from-python">pyplatypus: the same engine, from Python</h2>
<p>The engine is not hidden plumbing. <a href="https://github.com/maju116/pyplatypus" rel="nofollow" target="_blank">pyplatypus</a> is a package on PyPI in its own right &#8211; it too started life in 2022 on TensorFlow and was rewritten alongside &#8211; and if you work in Python you can use it directly:</p>
<div class="cell">
<pre>pip install &quot;pyplatypus[hub]&quot;</pre>
</div>
<p>The example below runs from both languages, out of one file.</p>
<p>The two are developed together, and in a fixed order. A feature lands in the engine first, the engine is published to PyPI, and only then does rplatypus move to it &#8211; each release of the R package pins one exact version of pyplatypus, so it cannot break because the engine changed underneath it. They also share one <a href="https://maju116.github.io/pyplatypus/config.html" rel="nofollow" target="_blank">configuration format</a>: the same YAML file describes the same experiment whichever language reads it, so an R user and a Python user can hand each other a run.</p>
<p>That is also how this series is written. The examples are in R, because that is where the gap is, and each post shows what the same step looks like from Python.</p>
</section>
<section id="the-gap" class="level2">
<h2 class="anchored" data-anchor-id="the-gap">The gap</h2>
<p>Here is the part that convinced me this was worth doing. R has excellent tools for <em>reading</em> medical images &#8211; RNifti, oro.dicom, divest &#8211; and a healthy deep learning stack. Almost nothing joins them. When I checked in September, the CRAN Task View for <a href="https://cran.r-project.org/view=MedicalImaging" rel="nofollow" target="_blank">Medical Image Analysis</a> (last updated August 2025) listed no deep learning packages at all.</p>
<p>So rplatypus is meant to be that missing piece: not a wrapper around a Python library you then have to learn, but an R package that happens to compute in Python.</p>
</section>
<section id="a-mask-without-training-anything" class="level2">
<h2 class="anchored" data-anchor-id="a-mask-without-training-anything">A mask, without training anything</h2>
<p>The quickest way to see whether a package does anything useful is to borrow a model someone already trained. rplatypus publishes weights by name &#8211; here a U-Net trained on the 2018 Data Science Bowl, the same nuclei dataset I used on this blog in 2020.</p>
<p>The folder <code>nuclei/</code> holds four microscopy images in the dataset’s own layout &#8211; one directory per image, with <code>images/</code> and <code>masks/</code> inside.</p>
<div class="cell">
<pre>library(rplatypus)

spec &lt;- platypus_spec(
  data = segmentation_data(&quot;nuclei&quot;, &quot;nuclei&quot;, colormap = binary_colormap),
  models = list(u_net(&quot;nuclei&quot;, input_shape = c(256, 256), blocks = 4, filters = 16,
                      weights = &quot;dsbowl-unet&quot;, fit = FALSE))
)
fit &lt;- platypus_fit(spec, verbose = FALSE)
masks &lt;- predict(fit, split = &quot;validation&quot;)
dim(masks)
#&gt; [1]   4 256 256</pre>
</div>
<p>Four images, 256 by 256 pixels, one class label per pixel. This ran on the processor, with no GPU; <code>platypus_device()</code> tells you whether yours will use one, and why not if it can’t.</p>
<div class="cell">
<pre>images &lt;- list.files(&quot;nuclei&quot;, pattern = &quot;[.]png$&quot;, recursive = TRUE, full.names = TRUE)
images &lt;- read_images(grep(&quot;/images/&quot;, images, value = TRUE), size = c(256, 256))
plot_masks(images, prediction = masks, colormap = binary_colormap)</pre>
<div class="cell-output-display">
<div id="fig-nuclei" class="quarto-float quarto-figure quarto-figure-center anchored">
<figure class="quarto-float quarto-float-fig figure">
<div aria-describedby="fig-nuclei-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
<img src="https://i1.wp.com/datascienceguts.com/posts/2026-11-03-rplatypus/index_files/figure-html/fig-nuclei-1.png?w=450&#038;ssl=1" class="img-fluid figure-img"  data-recalc-dims="1">
</div>
<figcaption class="quarto-float-caption-bottom quarto-float-caption quarto-float-fig" id="fig-nuclei-caption-0ceaefa1-69ba-4598-a22c-09a6ac19f8ca">
Figure 1: Nuclei found by the published <code>dsbowl-unet</code> weights, on four images of different kinds.
</figcaption>
</figure>
</div>
</div>
</div>
<p>The first three are fluorescence images, and the nuclei come out as clean blobs. The fourth is a brightfield image of stained cells, and the model finds specks - the small dark nuclei inside the cells. That is the hard kind of image in this dataset, and the next post looks at it closely.</p>
<p>There is no score here, and that is on purpose. These four images come from the same collection the weights were trained on, so any number measured on them would flatter the model. The numbers that mean something were measured on a held-out split when the weights were published, and <code>available_weights()</code> says what they are and what the model cannot do - for instance, that it is a <em>semantic</em> model, so touching nuclei come back as one region and a count taken from them would be wrong.</p>
</section>
<section id="the-same-thing-from-a-file" class="level2">
<h2 class="anchored" data-anchor-id="the-same-thing-from-a-file">The same thing from a file</h2>
<p>The same specification can live in a file. This one started as <code>save_spec(spec, &quot;nuclei.yaml&quot;)</code>, with the lists tidied by hand:</p>
<div class="code-with-filename">
<div class="code-with-filename-file">
<pre>nuclei.yaml</pre>
</div>
<pre>task: semantic_segmentation
data:
  train_path: nuclei
  validation_path: nuclei
  colormap: [[0, 0, 0], [255, 255, 255]]
models:
- name: nuclei
  architecture: u_net
  input_shape: [256, 256]
  blocks: 4
  filters: 16
  weights: dsbowl-unet
  fit: false</pre>
</div>
<p>Reading it back gives exactly the object built from arguments above:</p>
<div class="cell">
<pre>identical(as.list(platypus_spec(&quot;nuclei.yaml&quot;)), as.list(spec))
#&gt; [1] TRUE</pre>
</div>
<p>So <code>platypus_fit(platypus_spec(&quot;nuclei.yaml&quot;))</code> does everything the R code did, and nothing downstream can tell which way the specification was written. The Python engine reads the very same file:</p>
<div class="cell">
<pre>from pyplatypus import build_engine, from_yaml

engine = build_engine(from_yaml(&quot;nuclei.yaml&quot;))
engine.fit()  # with `fit: false` this loads the published weights instead of training
masks = engine.predict(&quot;nuclei&quot;, split=&quot;validation&quot;)</pre>
</div>
<p>A configuration file and an R script describe the same experiment, and either can be handed to a colleague - whichever language they work in.</p>
</section>
<section id="what-is-in-it" class="level2">
<h2 class="anchored" data-anchor-id="what-is-in-it">What is in it</h2>
<p>Briefly, because most of these get their own post:</p>
<ul>
<li><strong>Segmentation</strong>, 2D and 3D: U-Net, U-Net++, Res-U-Net and LinkNet, eleven losses, five metrics that can leave the background out - which matters when the background is most of the image.</li>
<li><strong>Classification</strong> - one label for a whole image, scored by AUC rather than accuracy, because a classifier that answers “no finding” where 3% have one scores 0.97 and has learnt nothing.</li>
<li><strong>Object detection</strong> with YOLOv3, with anchors fitted to your boxes rather than borrowed from COCO.</li>
<li><strong>Instance segmentation</strong> - a count of objects instead of an area.</li>
<li><strong>Volumes</strong>: NIfTI and DICOM in, with voxel spacing handled, a split by patient rather than by slice, and results in millilitres rather than voxels.</li>
<li><strong>A score per case, not one per dataset</strong>, so the worst image has a name.</li>
<li><strong>Refusals before a wasted day</strong>: a colormap that matches none of your masks, channels in an order you did not intend, a DICOM series with a missing slice - each stopped before training, by name.</li>
<li><strong>One specification</strong> - from R arguments or from a YAML file, read the same way by the Python engine.</li>
</ul>
<p>The full reference and seven vignettes, six of them precomputed from real runs, are at <a href="https://maju116.github.io/rplatypus/" rel="nofollow" target="_blank">maju116.github.io/rplatypus</a>.</p>
<p>This is where the package starts, not where it ends. The U-Net family, YOLOv3 and the classifier are the first models in it, not the only ones that will be: more architectures are coming, along with what is still missing - multi-label classification, ensembling, boxes in three dimensions. Which of those comes first depends partly on what people actually need, which is one more reason for the question at the end of this post.</p>
<p>Development is moving fast right now, and CRAN asks for new versions no more often than every month or two, so once the package is on CRAN, <code>install.packages(&quot;rplatypus&quot;)</code> will usually be a step behind. The GitHub version above will always be the newest.</p>
</section>
<section id="what-comes-next" class="level2">
<h2 class="anchored" data-anchor-id="what-comes-next">What comes next</h2>
<p>This is the first post of a series. Each of the next ones takes one dataset and one thing that measurement on it taught me:</p>
<ol type="1">
<li><strong>Nuclei, five years later</strong> - a model with a mean Dice of 0.855 that scores 0.006 on three images, and why the mean could not tell you.</li>
<li><strong>Classification</strong> - a classifier with 0.97 accuracy that has learnt nothing, and why balanced accuracy does not rescue it.</li>
<li><strong>Blood cells and YOLOv3</strong> - anchors, and why a point of mAP across five seeds is noise.</li>
<li><strong>CT volumes</strong> - a Dice of 0.88 and volumes 12-23% too large, which is not a contradiction.</li>
<li><strong>Retinal vessels</strong> - why you cannot resize a vessel nine pixels wide.</li>
<li><strong>Counting touching nuclei</strong> - 245 objects or 180, from the same forward pass.</li>
<li><strong>Pretrained encoders</strong> - measured on three modalities, where a 1.9M-parameter encoder beats ResNet-34 on two of them.</li>
</ol>
</section>
<section id="one-question-for-you" class="level2">
<h2 class="anchored" data-anchor-id="one-question-for-you">One question for you</h2>
<p>Everything above rests on an assumption I have not been able to test: that people do medical image analysis in R and would rather stay in R to do deep learning on it. If that is you - or if it is your lab, and you went to Python because there was nothing here - I would really like to hear about it. <a href="https://github.com/maju116/rplatypus/issues" rel="nofollow" target="_blank">Open an issue</a>, write to me at <a href="mailto:michalmaj116@gmail.com" rel="nofollow" target="_blank">michalmaj116@gmail.com</a>, or find me on <a href="https://www.linkedin.com/in/michal-maj116/" rel="nofollow" target="_blank">LinkedIn</a>.</p>
<p>And one more thing, since this is a good place to say it: I am looking for my next role. If your team works on computer vision, medical imaging, or any problem where a model has to be measured honestly before anyone trusts it, I would be glad to talk - the same email works.</p>


</section>

 
<div style="border: 1px solid; background: none repeat scroll 0 0 #EDEDED; margin: 1px; font-size: 13px;">
<div style="text-align: center;">To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceguts.com/posts/2026-11-03-rplatypus/"> Data Science Guts</a></strong>.</div>
<hr />
<a href="https://www.r-bloggers.com/" rel="nofollow">R-bloggers.com</a> offers <strong><a href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers" rel="nofollow">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" href="https://www.r-project.org/" rel="nofollow">R</a> news and tutorials about <a title="R tutorials" href="https://www.r-bloggers.com/how-to-learn-r-2/" rel="nofollow">learning R</a> and many other topics. <a title="Data science jobs" href="https://www.r-users.com/" rel="nofollow">Click here if you're looking to post or find an R/data-science job</a>.

<hr>Want to share your content on R-bloggers?<a href="https://www.r-bloggers.com/add-your-blog/" rel="nofollow"> click here</a> if you have a blog, or <a href="http://r-posts.com/" rel="nofollow"> here</a> if you don't.
</div><strong>Continue reading</strong>: <a href="https://www.r-bloggers.com/2026/11/rplatypus-rebuilt-deep-learning-for-medical-images-in-r/">rplatypus rebuilt: deep learning for medical images in R</a>]]></content:encoded>
					
		
		<enclosure url="" length="0" type="" />

		<post-id xmlns="com-wordpress:feed-additions:1">404628</post-id>	</item>
	</channel>
</rss>
